Published site at d5aaeee88b331e064830a2774f4fed238631457c.
[hbase-site.git] / book.html
1 <!DOCTYPE html>
2 <html lang="en">
3 <head>
4 <meta charset="UTF-8">
5 <!--[if IE]><meta http-equiv="X-UA-Compatible" content="IE=edge"><![endif]-->
6 <meta name="viewport" content="width=device-width, initial-scale=1.0">
7 <meta name="generator" content="Asciidoctor 1.5.5">
8 <meta name="author" content="Apache HBase Team">
9 <title>Apache HBase &#8482; Reference Guide</title>
10 <link rel="stylesheet" href="./hbase.css">
11 <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.6.3/css/font-awesome.min.css">
12 <link rel="stylesheet" href="./coderay-asciidoctor.css">
13 </head>
14 <body class="book toc2 toc-left">
15 <div id="header">
16 <h1>Apache HBase &#8482; Reference Guide</h1>
17 <div class="details">
18 <span id="author" class="author">Apache HBase Team</span><br>
19 <span id="email" class="email">&lt;<a href="mailto:hbase-dev@lists.apache.org">hbase-dev@lists.apache.org</a>&gt;</span><br>
20 <span id="revnumber">version 3.0.0-SNAPSHOT</span>
21 </div>
22 <div id="toc" class="toc2">
23 <div id="toctitle">Contents</div>
24 <ul class="sectlevel1">
25 <li><a href="#_preface">Preface</a></li>
26 <li><a href="#getting_started">Getting Started</a>
27 <ul class="sectlevel1">
28 <li><a href="#_introduction">1. Introduction</a></li>
29 <li><a href="#quickstart">2. Quick Start - Standalone HBase</a></li>
30 </ul>
31 </li>
32 <li><a href="#configuration">Apache HBase Configuration</a>
33 <ul class="sectlevel1">
34 <li><a href="#_configuration_files">3. Configuration Files</a></li>
35 <li><a href="#basic.prerequisites">4. Basic Prerequisites</a></li>
36 <li><a href="#standalone_dist">5. HBase run modes: Standalone and Distributed</a></li>
37 <li><a href="#confirm">6. Running and Confirming Your Installation</a></li>
38 <li><a href="#config.files">7. Default Configuration</a></li>
39 <li><a href="#example_config">8. Example Configurations</a></li>
40 <li><a href="#important_configurations">9. The Important Configurations</a></li>
41 <li><a href="#dyn_config">10. Dynamic Configuration</a></li>
42 </ul>
43 </li>
44 <li><a href="#upgrading">Upgrading</a>
45 <ul class="sectlevel1">
46 <li><a href="#hbase.versioning">11. HBase version number and compatibility</a></li>
47 <li><a href="#_rollback">12. Rollback</a></li>
48 <li><a href="#_upgrade_paths">13. Upgrade Paths</a></li>
49 </ul>
50 </li>
51 <li><a href="#shell">The Apache HBase Shell</a>
52 <ul class="sectlevel1">
53 <li><a href="#scripting">14. Scripting with Ruby</a></li>
54 <li><a href="#_running_the_shell_in_non_interactive_mode">15. Running the Shell in Non-Interactive Mode</a></li>
55 <li><a href="#hbase.shell.noninteractive">16. HBase Shell in OS Scripts</a></li>
56 <li><a href="#_read_hbase_shell_commands_from_a_command_file">17. Read HBase Shell Commands from a Command File</a></li>
57 <li><a href="#_passing_vm_options_to_the_shell">18. Passing VM Options to the Shell</a></li>
58 <li><a href="#_shell_tricks">19. Shell Tricks</a></li>
59 </ul>
60 </li>
61 <li><a href="#datamodel">Data Model</a>
62 <ul class="sectlevel1">
63 <li><a href="#conceptual.view">20. Conceptual View</a></li>
64 <li><a href="#physical.view">21. Physical View</a></li>
65 <li><a href="#_namespace">22. Namespace</a></li>
66 <li><a href="#_table">23. Table</a></li>
67 <li><a href="#_row">24. Row</a></li>
68 <li><a href="#columnfamily">25. Column Family</a></li>
69 <li><a href="#_cells">26. Cells</a></li>
70 <li><a href="#_data_model_operations">27. Data Model Operations</a></li>
71 <li><a href="#versions">28. Versions</a></li>
72 <li><a href="#dm.sort">29. Sort Order</a></li>
73 <li><a href="#dm.column.metadata">30. Column Metadata</a></li>
74 <li><a href="#joins">31. Joins</a></li>
75 <li><a href="#_acid">32. ACID</a></li>
76 </ul>
77 </li>
78 <li><a href="#schema">HBase and Schema Design</a>
79 <ul class="sectlevel1">
80 <li><a href="#schema.creation">33. Schema Creation</a></li>
81 <li><a href="#table_schema_rules_of_thumb">34. Table Schema Rules Of Thumb</a></li>
82 </ul>
83 </li>
84 <li><a href="#regionserver_sizing_rules_of_thumb">RegionServer Sizing Rules of Thumb</a>
85 <ul class="sectlevel1">
86 <li><a href="#number.of.cfs">35. On the number of column families</a></li>
87 <li><a href="#rowkey.design">36. Rowkey Design</a></li>
88 <li><a href="#schema.versions">37. Number of Versions</a></li>
89 <li><a href="#supported.datatypes">38. Supported Datatypes</a></li>
90 <li><a href="#schema.joins">39. Joins</a></li>
91 <li><a href="#ttl">40. Time To Live (TTL)</a></li>
92 <li><a href="#cf.keep.deleted">41. Keeping Deleted Cells</a></li>
93 <li><a href="#secondary.indexes">42. Secondary Indexes and Alternate Query Paths</a></li>
94 <li><a href="#_constraints">43. Constraints</a></li>
95 <li><a href="#schema.casestudies">44. Schema Design Case Studies</a></li>
96 <li><a href="#schema.ops">45. Operational and Performance Configuration Options</a></li>
97 <li><a href="#_special_cases">46. Special Cases</a></li>
98 </ul>
99 </li>
100 <li><a href="#mapreduce">HBase and MapReduce</a>
101 <ul class="sectlevel1">
102 <li><a href="#hbase.mapreduce.classpath">47. HBase, MapReduce, and the CLASSPATH</a></li>
103 <li><a href="#_mapreduce_scan_caching">48. MapReduce Scan Caching</a></li>
104 <li><a href="#_bundled_hbase_mapreduce_jobs">49. Bundled HBase MapReduce Jobs</a></li>
105 <li><a href="#_hbase_as_a_mapreduce_job_data_source_and_data_sink">50. HBase as a MapReduce Job Data Source and Data Sink</a></li>
106 <li><a href="#_writing_hfiles_directly_during_bulk_import">51. Writing HFiles Directly During Bulk Import</a></li>
107 <li><a href="#_rowcounter_example">52. RowCounter Example</a></li>
108 <li><a href="#splitter">53. Map-Task Splitting</a></li>
109 <li><a href="#mapreduce.example">54. HBase MapReduce Examples</a></li>
110 <li><a href="#mapreduce.htable.access">55. Accessing Other HBase Tables in a MapReduce Job</a></li>
111 <li><a href="#mapreduce.specex">56. Speculative Execution</a></li>
112 <li><a href="#cascading">57. Cascading</a></li>
113 </ul>
114 </li>
115 <li><a href="#security">Securing Apache HBase</a>
116 <ul class="sectlevel1">
117 <li><a href="#_using_secure_http_https_for_the_web_ui">58. Using Secure HTTP (HTTPS) for the Web UI</a></li>
118 <li><a href="#hbase.secure.spnego.ui">59. Using SPNEGO for Kerberos authentication with Web UIs</a></li>
119 <li><a href="#hbase.secure.configuration">60. Secure Client Access to Apache HBase</a></li>
120 <li><a href="#hbase.secure.simpleconfiguration">61. Simple User Access to Apache HBase</a></li>
121 <li><a href="#_securing_access_to_hdfs_and_zookeeper">62. Securing Access to HDFS and ZooKeeper</a></li>
122 <li><a href="#_securing_access_to_your_data">63. Securing Access To Your Data</a></li>
123 <li><a href="#security.example.config">64. Security Configuration Example</a></li>
124 </ul>
125 </li>
126 <li><a href="#_architecture">Architecture</a>
127 <ul class="sectlevel1">
128 <li><a href="#arch.overview">65. Overview</a></li>
129 <li><a href="#arch.catalog">66. Catalog Tables</a></li>
130 <li><a href="#architecture.client">67. Client</a></li>
131 <li><a href="#client.filter">68. Client Request Filters</a></li>
132 <li><a href="#architecture.master">69. Master</a></li>
133 <li><a href="#regionserver.arch">70. RegionServer</a></li>
134 <li><a href="#regions.arch">71. Regions</a></li>
135 <li><a href="#arch.bulk.load">72. Bulk Loading</a></li>
136 <li><a href="#arch.hdfs">73. HDFS</a></li>
137 <li><a href="#arch.timelineconsistent.reads">74. Timeline-consistent High Available Reads</a></li>
138 <li><a href="#hbase_mob">75. Storing Medium-sized Objects (MOB)</a></li>
139 </ul>
140 </li>
141 <li><a href="#casestudies">Backup and Restore</a>
142 <ul class="sectlevel1">
143 <li><a href="#br.overview">76. Overview</a></li>
144 <li><a href="#br.terminology">77. Terminology</a></li>
145 <li><a href="#br.planning">78. Planning</a></li>
146 <li><a href="#br.initial.setup">79. First-time configuration steps</a></li>
147 <li><a href="#_backup_and_restore_commands">80. Backup and Restore commands</a></li>
148 <li><a href="#br.administration">81. Administration of Backup Images</a></li>
149 <li><a href="#br.backup.configuration">82. Configuration keys</a></li>
150 <li><a href="#br.best.practices">83. Best Practices</a></li>
151 <li><a href="#br.s3.backup.scenario">84. Scenario: Safeguarding Application Datasets on Amazon S3</a></li>
152 <li><a href="#br.data.security">85. Security of Backup Data</a></li>
153 <li><a href="#br.technical.details">86. Technical Details of Incremental Backup and Restore</a></li>
154 <li><a href="#br.filesystem.growth.warning">87. A Warning on File System Growth</a></li>
155 <li><a href="#br.backup.capacity.planning">88. Capacity Planning</a></li>
156 <li><a href="#br.limitations">89. Limitations of the Backup and Restore Utility</a></li>
157 </ul>
158 </li>
159 <li><a href="#hbase_apis">Apache HBase APIs</a>
160 <ul class="sectlevel1">
161 <li><a href="#_examples">90. Examples</a></li>
162 </ul>
163 </li>
164 <li><a href="#external_apis">Apache HBase External APIs</a>
165 <ul class="sectlevel1">
166 <li><a href="#_rest">91. REST</a></li>
167 <li><a href="#_thrift">92. Thrift</a></li>
168 <li><a href="#c">93. C/C++ Apache HBase Client</a></li>
169 <li><a href="#jdo">94. Using Java Data Objects (JDO) with HBase</a></li>
170 <li><a href="#scala">95. Scala</a></li>
171 <li><a href="#jython">96. Jython</a></li>
172 </ul>
173 </li>
174 <li><a href="#thrift">Thrift API and Filter Language</a>
175 <ul class="sectlevel1">
176 <li><a href="#thrift.filter_language">97. Filter Language</a></li>
177 </ul>
178 </li>
179 <li><a href="#spark">HBase and Spark</a>
180 <ul class="sectlevel1">
181 <li><a href="#_basic_spark">98. Basic Spark</a></li>
182 <li><a href="#_spark_streaming">99. Spark Streaming</a></li>
183 <li><a href="#_bulk_load">100. Bulk Load</a></li>
184 <li><a href="#_sparksql_dataframes">101. SparkSQL/DataFrames</a></li>
185 </ul>
186 </li>
187 <li><a href="#cp">Apache HBase Coprocessors</a>
188 <ul class="sectlevel1">
189 <li><a href="#_coprocessor_overview">102. Coprocessor Overview</a></li>
190 <li><a href="#_types_of_coprocessors">103. Types of Coprocessors</a></li>
191 <li><a href="#cp_loading">104. Loading Coprocessors</a></li>
192 <li><a href="#cp_example">105. Examples</a></li>
193 <li><a href="#_guidelines_for_deploying_a_coprocessor">106. Guidelines For Deploying A Coprocessor</a></li>
194 <li><a href="#_restricting_coprocessor_usage">107. Restricting Coprocessor Usage</a></li>
195 </ul>
196 </li>
197 <li><a href="#performance">Apache HBase Performance Tuning</a>
198 <ul class="sectlevel1">
199 <li><a href="#perf.os">108. Operating System</a></li>
200 <li><a href="#perf.network">109. Network</a></li>
201 <li><a href="#jvm">110. Java</a></li>
202 <li><a href="#perf.configurations">111. HBase Configurations</a></li>
203 <li><a href="#perf.zookeeper">112. ZooKeeper</a></li>
204 <li><a href="#perf.schema">113. Schema Design</a></li>
205 <li><a href="#perf.general">114. HBase General Patterns</a></li>
206 <li><a href="#perf.writing">115. Writing to HBase</a></li>
207 <li><a href="#perf.reading">116. Reading from HBase</a></li>
208 <li><a href="#perf.deleting">117. Deleting from HBase</a></li>
209 <li><a href="#perf.hdfs">118. HDFS</a></li>
210 <li><a href="#perf.ec2">119. Amazon EC2</a></li>
211 <li><a href="#perf.hbase.mr.cluster">120. Collocating HBase and MapReduce</a></li>
212 <li><a href="#perf.casestudy">121. Case Studies</a></li>
213 </ul>
214 </li>
215 <li><a href="#trouble">Troubleshooting and Debugging Apache HBase</a>
216 <ul class="sectlevel1">
217 <li><a href="#trouble.general">122. General Guidelines</a></li>
218 <li><a href="#trouble.log">123. Logs</a></li>
219 <li><a href="#trouble.resources">124. Resources</a></li>
220 <li><a href="#trouble.tools">125. Tools</a></li>
221 <li><a href="#trouble.client">126. Client</a></li>
222 <li><a href="#trouble.mapreduce">127. MapReduce</a></li>
223 <li><a href="#trouble.namenode">128. NameNode</a></li>
224 <li><a href="#trouble.network">129. Network</a></li>
225 <li><a href="#trouble.rs">130. RegionServer</a></li>
226 <li><a href="#trouble.master">131. Master</a></li>
227 <li><a href="#trouble.zookeeper">132. ZooKeeper</a></li>
228 <li><a href="#trouble.ec2">133. Amazon EC2</a></li>
229 <li><a href="#trouble.versions">134. HBase and Hadoop version issues</a></li>
230 <li><a href="#_ipc_configuration_conflicts_with_hadoop">135. IPC Configuration Conflicts with Hadoop</a></li>
231 <li><a href="#_hbase_and_hdfs">136. HBase and HDFS</a></li>
232 <li><a href="#trouble.tests">137. Running unit or integration tests</a></li>
233 <li><a href="#trouble.casestudy">138. Case Studies</a></li>
234 <li><a href="#trouble.crypto">139. Cryptographic Features</a></li>
235 <li><a href="#_operating_system_specific_issues">140. Operating System Specific Issues</a></li>
236 <li><a href="#_jdk_issues">141. JDK Issues</a></li>
237 </ul>
238 </li>
239 <li><a href="#casestudies">Apache HBase Case Studies</a>
240 <ul class="sectlevel1">
241 <li><a href="#casestudies.overview">142. Overview</a></li>
242 <li><a href="#casestudies.schema">143. Schema Design</a></li>
243 <li><a href="#casestudies.perftroub">144. Performance/Troubleshooting</a></li>
244 </ul>
245 </li>
246 <li><a href="#ops_mgt">Apache HBase Operational Management</a>
247 <ul class="sectlevel1">
248 <li><a href="#tools">145. HBase Tools and Utilities</a></li>
249 <li><a href="#ops.regionmgt">146. Region Management</a></li>
250 <li><a href="#node.management">147. Node Management</a></li>
251 <li><a href="#hbase_metrics">148. HBase Metrics</a></li>
252 <li><a href="#ops.monitoring">149. HBase Monitoring</a></li>
253 <li><a href="#_cluster_replication">150. Cluster Replication</a></li>
254 <li><a href="#_running_multiple_workloads_on_a_single_cluster">151. Running Multiple Workloads On a Single Cluster</a></li>
255 <li><a href="#ops.backup">152. HBase Backup</a></li>
256 <li><a href="#ops.snapshots">153. HBase Snapshots</a></li>
257 <li><a href="#snapshots_azure">154. Storing Snapshots in Microsoft Azure Blob Storage</a></li>
258 <li><a href="#ops.capacity">155. Capacity Planning and Region Sizing</a></li>
259 <li><a href="#table.rename">156. Table Rename</a></li>
260 <li><a href="#rsgroup">157. RegionServer Grouping</a></li>
261 <li><a href="#normalizer">158. Region Normalizer</a></li>
262 </ul>
263 </li>
264 <li><a href="#developer">Building and Developing Apache HBase</a>
265 <ul class="sectlevel1">
266 <li><a href="#getting.involved">159. Getting Involved</a></li>
267 <li><a href="#repos">160. Apache HBase Repositories</a></li>
268 <li><a href="#_ides">161. IDEs</a></li>
269 <li><a href="#build">162. Building Apache HBase</a></li>
270 <li><a href="#releasing">163. Releasing Apache HBase</a></li>
271 <li><a href="#hbase.rc.voting">164. Voting on Release Candidates</a></li>
272 <li><a href="#documentation">165. Generating the HBase Reference Guide</a></li>
273 <li><a href="#hbase.org">166. Updating <a href="https://hbase.apache.org">hbase.apache.org</a></a></li>
274 <li><a href="#hbase.tests">167. Tests</a></li>
275 <li><a href="#developing">168. Developer Guidelines</a></li>
276 </ul>
277 </li>
278 <li><a href="#unit.tests">Unit Testing HBase Applications</a>
279 <ul class="sectlevel1">
280 <li><a href="#_junit">169. JUnit</a></li>
281 <li><a href="#mockito">170. Mockito</a></li>
282 <li><a href="#_mrunit">171. MRUnit</a></li>
283 <li><a href="#_integration_testing_with_an_hbase_mini_cluster">172. Integration Testing with an HBase Mini-Cluster</a></li>
284 </ul>
285 </li>
286 <li><a href="#protobuf">Protobuf in HBase</a>
287 <ul class="sectlevel1">
288 <li><a href="#_protobuf">173. Protobuf</a></li>
289 </ul>
290 </li>
291 <li><a href="#zookeeper">ZooKeeper</a>
292 <ul class="sectlevel1">
293 <li><a href="#_using_existing_zookeeper_ensemble">174. Using existing ZooKeeper ensemble</a></li>
294 <li><a href="#zk.sasl.auth">175. SASL Authentication with ZooKeeper</a></li>
295 </ul>
296 </li>
297 <li><a href="#community">Community</a>
298 <ul class="sectlevel1">
299 <li><a href="#_decisions">176. Decisions</a></li>
300 <li><a href="#community.roles">177. Community Roles</a></li>
301 <li><a href="#hbase.commit.msg.format">178. Commit Message format</a></li>
302 </ul>
303 </li>
304 <li><a href="#_appendix">Appendix</a>
305 <ul class="sectlevel1">
306 <li><a href="#appendix_contributing_to_documentation">Appendix A: Contributing to Documentation</a></li>
307 <li><a href="#faq">Appendix B: FAQ</a></li>
308 <li><a href="#hbck.in.depth">Appendix C: hbck In Depth</a></li>
309 <li><a href="#appendix_acl_matrix">Appendix D: Access Control Matrix</a></li>
310 <li><a href="#compression">Appendix E: Compression and Data Block Encoding In HBase</a></li>
311 <li><a href="#data.block.encoding.enable">179. Enable Data Block Encoding</a></li>
312 <li><a href="#sql">Appendix F: SQL over HBase</a></li>
313 <li><a href="#ycsb">Appendix G: YCSB</a></li>
314 <li><a href="#_hfile_format_2">Appendix H: HFile format</a></li>
315 <li><a href="#other.info">Appendix I: Other Information About HBase</a></li>
316 <li><a href="#hbase.history">Appendix J: HBase History</a></li>
317 <li><a href="#asf">Appendix K: HBase and the Apache Software Foundation</a></li>
318 <li><a href="#orca">Appendix L: Apache HBase Orca</a></li>
319 <li><a href="#tracing">Appendix M: Enabling Dapper-like Tracing in HBase</a></li>
320 <li><a href="#tracing.client.modifications">180. Client Modifications</a></li>
321 <li><a href="#tracing.client.shell">181. Tracing from HBase Shell</a></li>
322 <li><a href="#hbase.rpc">Appendix N: 0.95 RPC Specification</a></li>
323 </ul>
324 </li>
325 </ul>
326 </div>
327 </div>
328 <div id="content">
329 <div id="preamble">
330 <div class="sectionbody">
331 <div>
332 <a href="https://hbase.apache.org"><img src="images/hbase_logo_with_orca.png" alt="Apache HBase Logo" /></a>
333 </div>
334 </div>
335 </div>
336 <div class="sect1">
337 <h2 id="_preface"><a class="anchor" href="#_preface"></a>Preface</h2>
338 <div class="sectionbody">
339 <div class="paragraph">
340 <p>This is the official reference guide for the <a href="https://hbase.apache.org/">HBase</a> version it ships with.</p>
341 </div>
342 <div class="paragraph">
343 <p>Herein you will find either the definitive documentation on an HBase topic as of its
344 standing when the referenced HBase version shipped, or it will point to the location
345 in <a href="https://hbase.apache.org/apidocs/index.html">Javadoc</a> or
346 <a href="https://issues.apache.org/jira/browse/HBASE">JIRA</a> where the pertinent information can be found.</p>
347 </div>
348 <div class="paragraph">
349 <div class="title">About This Guide</div>
350 <p>This reference guide is a work in progress. The source for this guide can be found in the
351 _src/main/asciidoc directory of the HBase source. This reference guide is marked up
352 using <a href="http://asciidoc.org/">AsciiDoc</a> from which the finished guide is generated as part of the
353 'site' build target. Run</p>
354 </div>
355 <div class="listingblock">
356 <div class="content">
357 <pre class="CodeRay highlight"><code data-lang="bourne">mvn site</code></pre>
358 </div>
359 </div>
360 <div class="paragraph">
361 <p>to generate this documentation.
362 Amendments and improvements to the documentation are welcomed.
363 Click
364 <a href="https://issues.apache.org/jira/secure/CreateIssueDetails!init.jspa?pid=12310753&amp;issuetype=1&amp;components=12312132&amp;summary=SHORT+DESCRIPTION">this link</a>
365 to file a new documentation bug against Apache HBase with some values pre-selected.</p>
366 </div>
367 <div class="paragraph">
368 <div class="title">Contributing to the Documentation</div>
369 <p>For an overview of AsciiDoc and suggestions to get started contributing to the documentation,
370 see the <a href="#appendix_contributing_to_documentation">relevant section later in this documentation</a>.</p>
371 </div>
372 <div class="paragraph">
373 <div class="title">Heads-up if this is your first foray into the world of distributed computing&#8230;&#8203;</div>
374 <p>If this is your first foray into the wonderful world of Distributed Computing, then you are in for some interesting times.
375 First off, distributed systems are hard; making a distributed system hum requires a disparate skillset that spans systems (hardware and software) and networking.</p>
376 </div>
377 <div class="paragraph">
378 <p>Your cluster&#8217;s operation can hiccup because of any of a myriad set of reasons from bugs in HBase itself through misconfigurations&#8201;&#8212;&#8201;misconfiguration of HBase but also operating system misconfigurations&#8201;&#8212;&#8201;through to hardware problems whether it be a bug in your network card drivers or an underprovisioned RAM bus (to mention two recent examples of hardware issues that manifested as "HBase is slow"). You will also need to do a recalibration if up to this your computing has been bound to a single box.
379 Here is one good starting point: <a href="http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Computing">Fallacies of Distributed Computing</a>.</p>
380 </div>
381 <div class="paragraph">
382 <p>That said, you are welcome.<br>
383 It&#8217;s a fun place to be.<br>
384 Yours, the HBase Community.</p>
385 </div>
386 <div class="paragraph">
387 <div class="title">Reporting Bugs</div>
388 <p>Please use <a href="https://issues.apache.org/jira/browse/hbase">JIRA</a> to report non-security-related bugs.</p>
389 </div>
390 <div class="paragraph">
391 <p>To protect existing HBase installations from new vulnerabilities, please <strong>do not</strong> use JIRA to report security-related bugs. Instead, send your report to the mailing list <a href="mailto:private@apache.org">private@apache.org</a>, which allows anyone to send messages, but restricts who can read them. Someone on that list will contact you to follow up on your report.</p>
392 </div>
393 <div id="hbase_supported_tested_definitions" class="paragraph">
394 <div class="title">Support and Testing Expectations</div>
395 <p>The phrases /supported/, /not supported/, /tested/, and /not tested/ occur several
396 places throughout this guide. In the interest of clarity, here is a brief explanation
397 of what is generally meant by these phrases, in the context of HBase.</p>
398 </div>
399 <div class="admonitionblock note">
400 <table>
401 <tr>
402 <td class="icon">
403 <i class="fa icon-note" title="Note"></i>
404 </td>
405 <td class="content">
406 Commercial technical support for Apache HBase is provided by many Hadoop vendors.
407 This is not the sense in which the term /support/ is used in the context of the
408 Apache HBase project. The Apache HBase team assumes no responsibility for your
409 HBase clusters, your configuration, or your data.
410 </td>
411 </tr>
412 </table>
413 </div>
414 <div class="dlist">
415 <dl>
416 <dt class="hdlist1">Supported</dt>
417 <dd>
418 <p>In the context of Apache HBase, /supported/ means that HBase is designed to work
419 in the way described, and deviation from the defined behavior or functionality should
420 be reported as a bug.</p>
421 </dd>
422 <dt class="hdlist1">Not Supported</dt>
423 <dd>
424 <p>In the context of Apache HBase, /not supported/ means that a use case or use pattern
425 is not expected to work and should be considered an antipattern. If you think this
426 designation should be reconsidered for a given feature or use pattern, file a JIRA
427 or start a discussion on one of the mailing lists.</p>
428 </dd>
429 <dt class="hdlist1">Tested</dt>
430 <dd>
431 <p>In the context of Apache HBase, /tested/ means that a feature is covered by unit
432 or integration tests, and has been proven to work as expected.</p>
433 </dd>
434 <dt class="hdlist1">Not Tested</dt>
435 <dd>
436 <p>In the context of Apache HBase, /not tested/ means that a feature or use pattern
437 may or may not work in a given way, and may or may not corrupt your data or cause
438 operational issues. It is an unknown, and there are no guarantees. If you can provide
439 proof that a feature designated as /not tested/ does work in a given way, please
440 submit the tests and/or the metrics so that other users can gain certainty about
441 such features or use patterns.</p>
442 </dd>
443 </dl>
444 </div>
445 </div>
446 </div>
447 <h1 id="getting_started" class="sect0"><a class="anchor" href="#getting_started"></a>Getting Started</h1>
448 <div class="sect1">
449 <h2 id="_introduction"><a class="anchor" href="#_introduction"></a>1. Introduction</h2>
450 <div class="sectionbody">
451 <div class="paragraph">
452 <p><a href="#quickstart">Quickstart</a> will get you up and running on a single-node, standalone instance of HBase.</p>
453 </div>
454 </div>
455 </div>
456 <div class="sect1">
457 <h2 id="quickstart"><a class="anchor" href="#quickstart"></a>2. Quick Start - Standalone HBase</h2>
458 <div class="sectionbody">
459 <div class="paragraph">
460 <p>This section describes the setup of a single-node standalone HBase.
461 A <em>standalone</em> instance has all HBase daemons&#8201;&#8212;&#8201;the Master, RegionServers,
462 and ZooKeeper&#8201;&#8212;&#8201;running in a single JVM persisting to the local filesystem.
463 It is our most basic deploy profile. We will show you how
464 to create a table in HBase using the <code>hbase shell</code> CLI,
465 insert rows into the table, perform put and scan operations against the
466 table, enable or disable the table, and start and stop HBase.</p>
467 </div>
468 <div class="paragraph">
469 <p>Apart from downloading HBase, this procedure should take less than 10 minutes.</p>
470 </div>
471 <div id="loopback.ip" class="admonitionblock note">
472 <table>
473 <tr>
474 <td class="icon">
475 <i class="fa icon-note" title="Note"></i>
476 </td>
477 <td class="content">
478 <div class="paragraph">
479 <p>Prior to HBase 0.94.x, HBase expected the loopback IP address to be 127.0.0.1.
480 Ubuntu and some other distributions default to 127.0.1.1 and this will cause
481 problems for you. See <a href="https://web-beta.archive.org/web/20140104070155/http://blog.devving.com/why-does-hbase-care-about-etchosts">Why does HBase care about /etc/hosts?</a> for detail</p>
482 </div>
483 <div class="paragraph">
484 <p>The following <em>/etc/hosts</em> file works correctly for HBase 0.94.x and earlier, on Ubuntu. Use this as a template if you run into trouble.</p>
485 </div>
486 <div class="listingblock">
487 <div class="content">
488 <pre>127.0.0.1 localhost
489 127.0.0.1 ubuntu.ubuntu-domain ubuntu</pre>
490 </div>
491 </div>
492 <div class="paragraph">
493 <p>This issue has been fixed in hbase-0.96.0 and beyond.</p>
494 </div>
495 </td>
496 </tr>
497 </table>
498 </div>
499 <div class="sect2">
500 <h3 id="_jdk_version_requirements"><a class="anchor" href="#_jdk_version_requirements"></a>2.1. JDK Version Requirements</h3>
501 <div class="paragraph">
502 <p>HBase requires that a JDK be installed.
503 See <a href="#java">Java</a> for information about supported JDK versions.</p>
504 </div>
505 </div>
506 <div class="sect2">
507 <h3 id="_get_started_with_hbase"><a class="anchor" href="#_get_started_with_hbase"></a>2.2. Get Started with HBase</h3>
508 <div class="olist arabic">
509 <div class="title">Procedure: Download, Configure, and Start HBase in Standalone Mode</div>
510 <ol class="arabic">
511 <li>
512 <p>Choose a download site from this list of <a href="https://www.apache.org/dyn/closer.cgi/hbase/">Apache Download Mirrors</a>.
513 Click on the suggested top link.
514 This will take you to a mirror of <em>HBase Releases</em>.
515 Click on the folder named <em>stable</em> and then download the binary file that ends in <em>.tar.gz</em> to your local filesystem.
516 Do not download the file ending in <em>src.tar.gz</em> for now.</p>
517 </li>
518 <li>
519 <p>Extract the downloaded file, and change to the newly-created directory.</p>
520 <div class="listingblock">
521 <div class="content">
522 <pre class="CodeRay highlight"><code data-lang="java">$ tar xzvf hbase-3.0.0-SNAPSHOT-bin.tar.gz
523 $ cd hbase-3.0.0-SNAPSHOT/</code></pre>
524 </div>
525 </div>
526 </li>
527 <li>
528 <p>You are required to set the <code>JAVA_HOME</code> environment variable before starting HBase.
529 You can set the variable via your operating system&#8217;s usual mechanism, but HBase
530 provides a central mechanism, <em>conf/hbase-env.sh</em>.
531 Edit this file, uncomment the line starting with <code>JAVA_HOME</code>, and set it to the
532 appropriate location for your operating system.
533 The <code>JAVA_HOME</code> variable should be set to a directory which contains the executable file <em>bin/java</em>.
534 Most modern Linux operating systems provide a mechanism, such as /usr/bin/alternatives on RHEL or CentOS, for transparently switching between versions of executables such as Java.
535 In this case, you can set <code>JAVA_HOME</code> to the directory containing the symbolic link to <em>bin/java</em>, which is usually <em>/usr</em>.</p>
536 <div class="listingblock">
537 <div class="content">
538 <pre>JAVA_HOME=/usr</pre>
539 </div>
540 </div>
541 </li>
542 <li>
543 <p>Edit <em>conf/hbase-site.xml</em>, which is the main HBase configuration file.
544 At this time, you only need to specify the directory on the local filesystem where HBase and ZooKeeper write data.
545 By default, a new directory is created under /tmp.
546 Many servers are configured to delete the contents of <em>/tmp</em> upon reboot, so you should store the data elsewhere.
547 The following configuration will store HBase&#8217;s data in the <em>hbase</em> directory, in the home directory of the user called <code>testuser</code>.
548 Paste the <code>&lt;property&gt;</code> tags beneath the <code>&lt;configuration&gt;</code> tags, which should be empty in a new HBase install.</p>
549 <div class="exampleblock">
550 <div class="title">Example 1. Example <em>hbase-site.xml</em> for Standalone HBase</div>
551 <div class="content">
552 <div class="listingblock">
553 <div class="content">
554 <pre class="CodeRay highlight"><code data-lang="xml"><span class="tag">&lt;configuration&gt;</span>
555 <span class="tag">&lt;property&gt;</span>
556 <span class="tag">&lt;name&gt;</span>hbase.rootdir<span class="tag">&lt;/name&gt;</span>
557 <span class="tag">&lt;value&gt;</span>file:///home/testuser/hbase<span class="tag">&lt;/value&gt;</span>
558 <span class="tag">&lt;/property&gt;</span>
559 <span class="tag">&lt;property&gt;</span>
560 <span class="tag">&lt;name&gt;</span>hbase.zookeeper.property.dataDir<span class="tag">&lt;/name&gt;</span>
561 <span class="tag">&lt;value&gt;</span>/home/testuser/zookeeper<span class="tag">&lt;/value&gt;</span>
562 <span class="tag">&lt;/property&gt;</span>
563 <span class="tag">&lt;/configuration&gt;</span></code></pre>
564 </div>
565 </div>
566 </div>
567 </div>
568 <div class="paragraph">
569 <p>You do not need to create the HBase data directory.
570 HBase will do this for you. If you create the directory,
571 HBase will attempt to do a migration, which is not what you want.</p>
572 </div>
573 <div class="admonitionblock note">
574 <table>
575 <tr>
576 <td class="icon">
577 <i class="fa icon-note" title="Note"></i>
578 </td>
579 <td class="content">
580 The <em>hbase.rootdir</em> in the above example points to a directory
581 in the <em>local filesystem</em>. The 'file:/' prefix is how we denote local filesystem.
582 To home HBase on an existing instance of HDFS, set the <em>hbase.rootdir</em> to point at a
583 directory up on your instance: e.g. <em>hdfs://namenode.example.org:8020/hbase</em>.
584 For more on this variant, see the section below on Standalone HBase over HDFS.
585 </td>
586 </tr>
587 </table>
588 </div>
589 </li>
590 <li>
591 <p>The <em>bin/start-hbase.sh</em> script is provided as a convenient way to start HBase.
592 Issue the command, and if all goes well, a message is logged to standard output showing that HBase started successfully.
593 You can use the <code>jps</code> command to verify that you have one running process called <code>HMaster</code>.
594 In standalone mode HBase runs all daemons within this single JVM, i.e.
595 the HMaster, a single HRegionServer, and the ZooKeeper daemon.
596 Go to <em><a href="http://localhost:16010" class="bare">http://localhost:16010</a></em> to view the HBase Web UI.</p>
597 <div class="admonitionblock note">
598 <table>
599 <tr>
600 <td class="icon">
601 <i class="fa icon-note" title="Note"></i>
602 </td>
603 <td class="content">
604 Java needs to be installed and available.
605 If you get an error indicating that Java is not installed,
606 but it is on your system, perhaps in a non-standard location,
607 edit the <em>conf/hbase-env.sh</em> file and modify the <code>JAVA_HOME</code>
608 setting to point to the directory that contains <em>bin/java</em> on your system.
609 </td>
610 </tr>
611 </table>
612 </div>
613 </li>
614 </ol>
615 </div>
616 <div id="shell_exercises" class="olist arabic">
617 <div class="title">Procedure: Use HBase For the First Time</div>
618 <ol class="arabic">
619 <li>
620 <p>Connect to HBase.</p>
621 <div class="paragraph">
622 <p>Connect to your running instance of HBase using the <code>hbase shell</code> command, located in the <em class="path">bin/</em> directory of your HBase install.
623 In this example, some usage and version information that is printed when you start HBase Shell has been omitted.
624 The HBase Shell prompt ends with a <code>&gt;</code> character.</p>
625 </div>
626 <div class="listingblock">
627 <div class="content">
628 <pre>$ ./bin/hbase shell
629 hbase(main):001:0&gt;</pre>
630 </div>
631 </div>
632 </li>
633 <li>
634 <p>Display HBase Shell Help Text.</p>
635 <div class="paragraph">
636 <p>Type <code>help</code> and press Enter, to display some basic usage information for HBase Shell, as well as several example commands.
637 Notice that table names, rows, columns all must be enclosed in quote characters.</p>
638 </div>
639 </li>
640 <li>
641 <p>Create a table.</p>
642 <div class="paragraph">
643 <p>Use the <code>create</code> command to create a new table.
644 You must specify the table name and the ColumnFamily name.</p>
645 </div>
646 <div class="listingblock">
647 <div class="content">
648 <pre>hbase(main):001:0&gt; create 'test', 'cf'
649 0 row(s) in 0.4170 seconds
650
651 =&gt; Hbase::Table - test</pre>
652 </div>
653 </div>
654 </li>
655 <li>
656 <p>List Information About your Table</p>
657 <div class="paragraph">
658 <p>Use the <code>list</code> command to</p>
659 </div>
660 <div class="listingblock">
661 <div class="content">
662 <pre>hbase(main):002:0&gt; list 'test'
663 TABLE
664 test
665 1 row(s) in 0.0180 seconds
666
667 =&gt; ["test"]</pre>
668 </div>
669 </div>
670 </li>
671 <li>
672 <p>Put data into your table.</p>
673 <div class="paragraph">
674 <p>To put data into your table, use the <code>put</code> command.</p>
675 </div>
676 <div class="listingblock">
677 <div class="content">
678 <pre>hbase(main):003:0&gt; put 'test', 'row1', 'cf:a', 'value1'
679 0 row(s) in 0.0850 seconds
680
681 hbase(main):004:0&gt; put 'test', 'row2', 'cf:b', 'value2'
682 0 row(s) in 0.0110 seconds
683
684 hbase(main):005:0&gt; put 'test', 'row3', 'cf:c', 'value3'
685 0 row(s) in 0.0100 seconds</pre>
686 </div>
687 </div>
688 <div class="paragraph">
689 <p>Here, we insert three values, one at a time.
690 The first insert is at <code>row1</code>, column <code>cf:a</code>, with a value of <code>value1</code>.
691 Columns in HBase are comprised of a column family prefix, <code>cf</code> in this example, followed by a colon and then a column qualifier suffix, <code>a</code> in this case.</p>
692 </div>
693 </li>
694 <li>
695 <p>Scan the table for all data at once.</p>
696 <div class="paragraph">
697 <p>One of the ways to get data from HBase is to scan.
698 Use the <code>scan</code> command to scan the table for data.
699 You can limit your scan, but for now, all data is fetched.</p>
700 </div>
701 <div class="listingblock">
702 <div class="content">
703 <pre>hbase(main):006:0&gt; scan 'test'
704 ROW COLUMN+CELL
705 row1 column=cf:a, timestamp=1421762485768, value=value1
706 row2 column=cf:b, timestamp=1421762491785, value=value2
707 row3 column=cf:c, timestamp=1421762496210, value=value3
708 3 row(s) in 0.0230 seconds</pre>
709 </div>
710 </div>
711 </li>
712 <li>
713 <p>Get a single row of data.</p>
714 <div class="paragraph">
715 <p>To get a single row of data at a time, use the <code>get</code> command.</p>
716 </div>
717 <div class="listingblock">
718 <div class="content">
719 <pre>hbase(main):007:0&gt; get 'test', 'row1'
720 COLUMN CELL
721 cf:a timestamp=1421762485768, value=value1
722 1 row(s) in 0.0350 seconds</pre>
723 </div>
724 </div>
725 </li>
726 <li>
727 <p>Disable a table.</p>
728 <div class="paragraph">
729 <p>If you want to delete a table or change its settings, as well as in some other situations, you need to disable the table first, using the <code>disable</code> command.
730 You can re-enable it using the <code>enable</code> command.</p>
731 </div>
732 <div class="listingblock">
733 <div class="content">
734 <pre>hbase(main):008:0&gt; disable 'test'
735 0 row(s) in 1.1820 seconds
736
737 hbase(main):009:0&gt; enable 'test'
738 0 row(s) in 0.1770 seconds</pre>
739 </div>
740 </div>
741 <div class="paragraph">
742 <p>Disable the table again if you tested the <code>enable</code> command above:</p>
743 </div>
744 <div class="listingblock">
745 <div class="content">
746 <pre>hbase(main):010:0&gt; disable 'test'
747 0 row(s) in 1.1820 seconds</pre>
748 </div>
749 </div>
750 </li>
751 <li>
752 <p>Drop the table.</p>
753 <div class="paragraph">
754 <p>To drop (delete) a table, use the <code>drop</code> command.</p>
755 </div>
756 <div class="listingblock">
757 <div class="content">
758 <pre>hbase(main):011:0&gt; drop 'test'
759 0 row(s) in 0.1370 seconds</pre>
760 </div>
761 </div>
762 </li>
763 <li>
764 <p>Exit the HBase Shell.</p>
765 <div class="paragraph">
766 <p>To exit the HBase Shell and disconnect from your cluster, use the <code>quit</code> command.
767 HBase is still running in the background.</p>
768 </div>
769 </li>
770 </ol>
771 </div>
772 <div class="olist arabic">
773 <div class="title">Procedure: Stop HBase</div>
774 <ol class="arabic">
775 <li>
776 <p>In the same way that the <em>bin/start-hbase.sh</em> script is provided to conveniently start all HBase daemons, the <em>bin/stop-hbase.sh</em> script stops them.</p>
777 <div class="listingblock">
778 <div class="content">
779 <pre>$ ./bin/stop-hbase.sh
780 stopping hbase....................
781 $</pre>
782 </div>
783 </div>
784 </li>
785 <li>
786 <p>After issuing the command, it can take several minutes for the processes to shut down.
787 Use the <code>jps</code> to be sure that the HMaster and HRegionServer processes are shut down.</p>
788 </li>
789 </ol>
790 </div>
791 <div class="paragraph">
792 <p>The above has shown you how to start and stop a standalone instance of HBase.
793 In the next sections we give a quick overview of other modes of hbase deploy.</p>
794 </div>
795 </div>
796 <div class="sect2">
797 <h3 id="quickstart_pseudo"><a class="anchor" href="#quickstart_pseudo"></a>2.3. Pseudo-Distributed Local Install</h3>
798 <div class="paragraph">
799 <p>After working your way through <a href="#quickstart">quickstart</a> standalone mode,
800 you can re-configure HBase to run in pseudo-distributed mode.
801 Pseudo-distributed mode means that HBase still runs completely on a single host,
802 but each HBase daemon (HMaster, HRegionServer, and ZooKeeper) runs as a separate process:
803 in standalone mode all daemons ran in one jvm process/instance.
804 By default, unless you configure the <code>hbase.rootdir</code> property as described in
805 <a href="#quickstart">quickstart</a>, your data is still stored in <em>/tmp/</em>.
806 In this walk-through, we store your data in HDFS instead, assuming you have HDFS available.
807 You can skip the HDFS configuration to continue storing your data in the local filesystem.</p>
808 </div>
809 <div class="admonitionblock note">
810 <table>
811 <tr>
812 <td class="icon">
813 <i class="fa icon-note" title="Note"></i>
814 </td>
815 <td class="content">
816 <div class="title">Hadoop Configuration</div>
817 <div class="paragraph">
818 <p>This procedure assumes that you have configured Hadoop and HDFS on your local system and/or a remote
819 system, and that they are running and available. It also assumes you are using Hadoop 2.
820 The guide on
821 <a href="https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html">Setting up a Single Node Cluster</a>
822 in the Hadoop documentation is a good starting point.</p>
823 </div>
824 </td>
825 </tr>
826 </table>
827 </div>
828 <div class="olist arabic">
829 <ol class="arabic">
830 <li>
831 <p>Stop HBase if it is running.</p>
832 <div class="paragraph">
833 <p>If you have just finished <a href="#quickstart">quickstart</a> and HBase is still running, stop it.
834 This procedure will create a totally new directory where HBase will store its data, so any databases you created before will be lost.</p>
835 </div>
836 </li>
837 <li>
838 <p>Configure HBase.</p>
839 <div class="paragraph">
840 <p>Edit the <em>hbase-site.xml</em> configuration.
841 First, add the following property which directs HBase to run in distributed mode, with one JVM instance per daemon.</p>
842 </div>
843 <div class="listingblock">
844 <div class="content">
845 <pre class="CodeRay highlight"><code data-lang="xml"><span class="tag">&lt;property&gt;</span>
846 <span class="tag">&lt;name&gt;</span>hbase.cluster.distributed<span class="tag">&lt;/name&gt;</span>
847 <span class="tag">&lt;value&gt;</span>true<span class="tag">&lt;/value&gt;</span>
848 <span class="tag">&lt;/property&gt;</span></code></pre>
849 </div>
850 </div>
851 <div class="paragraph">
852 <p>Next, change the <code>hbase.rootdir</code> from the local filesystem to the address of your HDFS instance, using the <code>hdfs:////</code> URI syntax.
853 In this example, HDFS is running on the localhost at port 8020.</p>
854 </div>
855 <div class="listingblock">
856 <div class="content">
857 <pre class="CodeRay highlight"><code data-lang="xml"><span class="tag">&lt;property&gt;</span>
858 <span class="tag">&lt;name&gt;</span>hbase.rootdir<span class="tag">&lt;/name&gt;</span>
859 <span class="tag">&lt;value&gt;</span>hdfs://localhost:8020/hbase<span class="tag">&lt;/value&gt;</span>
860 <span class="tag">&lt;/property&gt;</span></code></pre>
861 </div>
862 </div>
863 <div class="paragraph">
864 <p>You do not need to create the directory in HDFS.
865 HBase will do this for you.
866 If you create the directory, HBase will attempt to do a migration, which is not what you want.</p>
867 </div>
868 </li>
869 <li>
870 <p>Start HBase.</p>
871 <div class="paragraph">
872 <p>Use the <em>bin/start-hbase.sh</em> command to start HBase.
873 If your system is configured correctly, the <code>jps</code> command should show the HMaster and HRegionServer processes running.</p>
874 </div>
875 </li>
876 <li>
877 <p>Check the HBase directory in HDFS.</p>
878 <div class="paragraph">
879 <p>If everything worked correctly, HBase created its directory in HDFS.
880 In the configuration above, it is stored in <em>/hbase/</em> on HDFS.
881 You can use the <code>hadoop fs</code> command in Hadoop&#8217;s <em>bin/</em> directory to list this directory.</p>
882 </div>
883 <div class="listingblock">
884 <div class="content">
885 <pre>$ ./bin/hadoop fs -ls /hbase
886 Found 7 items
887 drwxr-xr-x - hbase users 0 2014-06-25 18:58 /hbase/.tmp
888 drwxr-xr-x - hbase users 0 2014-06-25 21:49 /hbase/WALs
889 drwxr-xr-x - hbase users 0 2014-06-25 18:48 /hbase/corrupt
890 drwxr-xr-x - hbase users 0 2014-06-25 18:58 /hbase/data
891 -rw-r--r-- 3 hbase users 42 2014-06-25 18:41 /hbase/hbase.id
892 -rw-r--r-- 3 hbase users 7 2014-06-25 18:41 /hbase/hbase.version
893 drwxr-xr-x - hbase users 0 2014-06-25 21:49 /hbase/oldWALs</pre>
894 </div>
895 </div>
896 </li>
897 <li>
898 <p>Create a table and populate it with data.</p>
899 <div class="paragraph">
900 <p>You can use the HBase Shell to create a table, populate it with data, scan and get values from it, using the same procedure as in <a href="#shell_exercises">shell exercises</a>.</p>
901 </div>
902 </li>
903 <li>
904 <p>Start and stop a backup HBase Master (HMaster) server.</p>
905 <div class="admonitionblock note">
906 <table>
907 <tr>
908 <td class="icon">
909 <i class="fa icon-note" title="Note"></i>
910 </td>
911 <td class="content">
912 Running multiple HMaster instances on the same hardware does not make sense in a production environment, in the same way that running a pseudo-distributed cluster does not make sense for production.
913 This step is offered for testing and learning purposes only.
914 </td>
915 </tr>
916 </table>
917 </div>
918 <div class="paragraph">
919 <p>The HMaster server controls the HBase cluster.
920 You can start up to 9 backup HMaster servers, which makes 10 total HMasters, counting the primary.
921 To start a backup HMaster, use the <code>local-master-backup.sh</code>.
922 For each backup master you want to start, add a parameter representing the port offset for that master.
923 Each HMaster uses three ports (16010, 16020, and 16030 by default). The port offset is added to these ports, so using an offset of 2, the backup HMaster would use ports 16012, 16022, and 16032.
924 The following command starts 3 backup servers using ports 16012/16022/16032, 16013/16023/16033, and 16015/16025/16035.</p>
925 </div>
926 <div class="listingblock">
927 <div class="content">
928 <pre>$ ./bin/local-master-backup.sh 2 3 5</pre>
929 </div>
930 </div>
931 <div class="paragraph">
932 <p>To kill a backup master without killing the entire cluster, you need to find its process ID (PID). The PID is stored in a file with a name like <em>/tmp/hbase-USER-X-master.pid</em>.
933 The only contents of the file is the PID.
934 You can use the <code>kill -9</code> command to kill that PID.
935 The following command will kill the master with port offset 1, but leave the cluster running:</p>
936 </div>
937 <div class="listingblock">
938 <div class="content">
939 <pre>$ cat /tmp/hbase-testuser-1-master.pid |xargs kill -9</pre>
940 </div>
941 </div>
942 </li>
943 <li>
944 <p>Start and stop additional RegionServers</p>
945 <div class="paragraph">
946 <p>The HRegionServer manages the data in its StoreFiles as directed by the HMaster.
947 Generally, one HRegionServer runs per node in the cluster.
948 Running multiple HRegionServers on the same system can be useful for testing in pseudo-distributed mode.
949 The <code>local-regionservers.sh</code> command allows you to run multiple RegionServers.
950 It works in a similar way to the <code>local-master-backup.sh</code> command, in that each parameter you provide represents the port offset for an instance.
951 Each RegionServer requires two ports, and the default ports are 16020 and 16030.
952 However, the base ports for additional RegionServers are not the default ports since the default ports are used by the HMaster, which is also a RegionServer since HBase version 1.0.0.
953 The base ports are 16200 and 16300 instead.
954 You can run 99 additional RegionServers that are not a HMaster or backup HMaster, on a server.
955 The following command starts four additional RegionServers, running on sequential ports starting at 16202/16302 (base ports 16200/16300 plus 2).</p>
956 </div>
957 <div class="listingblock">
958 <div class="content">
959 <pre>$ .bin/local-regionservers.sh start 2 3 4 5</pre>
960 </div>
961 </div>
962 <div class="paragraph">
963 <p>To stop a RegionServer manually, use the <code>local-regionservers.sh</code> command with the <code>stop</code> parameter and the offset of the server to stop.</p>
964 </div>
965 <div class="listingblock">
966 <div class="content">
967 <pre>$ .bin/local-regionservers.sh stop 3</pre>
968 </div>
969 </div>
970 </li>
971 <li>
972 <p>Stop HBase.</p>
973 <div class="paragraph">
974 <p>You can stop HBase the same way as in the <a href="#quickstart">quickstart</a> procedure, using the <em>bin/stop-hbase.sh</em> command.</p>
975 </div>
976 </li>
977 </ol>
978 </div>
979 </div>
980 <div class="sect2">
981 <h3 id="quickstart_fully_distributed"><a class="anchor" href="#quickstart_fully_distributed"></a>2.4. Advanced - Fully Distributed</h3>
982 <div class="paragraph">
983 <p>In reality, you need a fully-distributed configuration to fully test HBase and to use it in real-world scenarios.
984 In a distributed configuration, the cluster contains multiple nodes, each of which runs one or more HBase daemon.
985 These include primary and backup Master instances, multiple ZooKeeper nodes, and multiple RegionServer nodes.</p>
986 </div>
987 <div class="paragraph">
988 <p>This advanced quickstart adds two more nodes to your cluster.
989 The architecture will be as follows:</p>
990 </div>
991 <table class="tableblock frame-all grid-all spread">
992 <caption class="title">Table 1. Distributed Cluster Demo Architecture</caption>
993 <colgroup>
994 <col style="width: 25%;">
995 <col style="width: 25%;">
996 <col style="width: 25%;">
997 <col style="width: 25%;">
998 </colgroup>
999 <thead>
1000 <tr>
1001 <th class="tableblock halign-left valign-top">Node Name</th>
1002 <th class="tableblock halign-left valign-top">Master</th>
1003 <th class="tableblock halign-left valign-top">ZooKeeper</th>
1004 <th class="tableblock halign-left valign-top">RegionServer</th>
1005 </tr>
1006 </thead>
1007 <tbody>
1008 <tr>
1009 <td class="tableblock halign-left valign-top"><p class="tableblock">node-a.example.com</p></td>
1010 <td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
1011 <td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
1012 <td class="tableblock halign-left valign-top"><p class="tableblock">no</p></td>
1013 </tr>
1014 <tr>
1015 <td class="tableblock halign-left valign-top"><p class="tableblock">node-b.example.com</p></td>
1016 <td class="tableblock halign-left valign-top"><p class="tableblock">backup</p></td>
1017 <td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
1018 <td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
1019 </tr>
1020 <tr>
1021 <td class="tableblock halign-left valign-top"><p class="tableblock">node-c.example.com</p></td>
1022 <td class="tableblock halign-left valign-top"><p class="tableblock">no</p></td>
1023 <td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
1024 <td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
1025 </tr>
1026 </tbody>
1027 </table>
1028 <div class="paragraph">
1029 <p>This quickstart assumes that each node is a virtual machine and that they are all on the same network.
1030 It builds upon the previous quickstart, <a href="#quickstart_pseudo">Pseudo-Distributed Local Install</a>, assuming that the system you configured in that procedure is now <code>node-a</code>.
1031 Stop HBase on <code>node-a</code> before continuing.</p>
1032 </div>
1033 <div class="admonitionblock note">
1034 <table>
1035 <tr>
1036 <td class="icon">
1037 <i class="fa icon-note" title="Note"></i>
1038 </td>
1039 <td class="content">
1040 Be sure that all the nodes have full access to communicate, and that no firewall rules are in place which could prevent them from talking to each other.
1041 If you see any errors like <code>no route to host</code>, check your firewall.
1042 </td>
1043 </tr>
1044 </table>
1045 </div>
1046 <div id="passwordless.ssh.quickstart" class="paragraph">
1047 <div class="title">Procedure: Configure Passwordless SSH Access</div>
1048 <p><code>node-a</code> needs to be able to log into <code>node-b</code> and <code>node-c</code> (and to itself) in order to start the daemons.
1049 The easiest way to accomplish this is to use the same username on all hosts, and configure password-less SSH login from <code>node-a</code> to each of the others.</p>
1050 </div>
1051 <div class="olist arabic">
1052 <ol class="arabic">
1053 <li>
1054 <p>On <code>node-a</code>, generate a key pair.</p>
1055 <div class="paragraph">
1056 <p>While logged in as the user who will run HBase, generate a SSH key pair, using the following command:</p>
1057 </div>
1058 <div class="listingblock">
1059 <div class="content">
1060 <pre class="CodeRay highlight"><code data-lang="bash">$ ssh-keygen -t rsa</code></pre>
1061 </div>
1062 </div>
1063 <div class="paragraph">
1064 <p>If the command succeeds, the location of the key pair is printed to standard output.
1065 The default name of the public key is <em>id_rsa.pub</em>.</p>
1066 </div>
1067 </li>
1068 <li>
1069 <p>Create the directory that will hold the shared keys on the other nodes.</p>
1070 <div class="paragraph">
1071 <p>On <code>node-b</code> and <code>node-c</code>, log in as the HBase user and create a <em>.ssh/</em> directory in the user&#8217;s home directory, if it does not already exist.
1072 If it already exists, be aware that it may already contain other keys.</p>
1073 </div>
1074 </li>
1075 <li>
1076 <p>Copy the public key to the other nodes.</p>
1077 <div class="paragraph">
1078 <p>Securely copy the public key from <code>node-a</code> to each of the nodes, by using the <code>scp</code> or some other secure means.
1079 On each of the other nodes, create a new file called <em>.ssh/authorized_keys</em> <em>if it does
1080 not already exist</em>, and append the contents of the <em>id_rsa.pub</em> file to the end of it.
1081 Note that you also need to do this for <code>node-a</code> itself.</p>
1082 </div>
1083 <div class="listingblock">
1084 <div class="content">
1085 <pre>$ cat id_rsa.pub &gt;&gt; ~/.ssh/authorized_keys</pre>
1086 </div>
1087 </div>
1088 </li>
1089 <li>
1090 <p>Test password-less login.</p>
1091 <div class="paragraph">
1092 <p>If you performed the procedure correctly, you should not be prompted for a password when you SSH from <code>node-a</code> to either of the other nodes using the same username.</p>
1093 </div>
1094 </li>
1095 <li>
1096 <p>Since <code>node-b</code> will run a backup Master, repeat the procedure above, substituting <code>node-b</code> everywhere you see <code>node-a</code>.
1097 Be sure not to overwrite your existing <em>.ssh/authorized_keys</em> files, but concatenate the new key onto the existing file using the <code>&gt;&gt;</code> operator rather than the <code>&gt;</code> operator.</p>
1098 </li>
1099 </ol>
1100 </div>
1101 <div class="paragraph">
1102 <div class="title">Procedure: Prepare <code>node-a</code></div>
1103 <p><code>node-a</code> will run your primary master and ZooKeeper processes, but no RegionServers. Stop the RegionServer from starting on <code>node-a</code>.</p>
1104 </div>
1105 <div class="olist arabic">
1106 <ol class="arabic">
1107 <li>
1108 <p>Edit <em>conf/regionservers</em> and remove the line which contains <code>localhost</code>. Add lines with the hostnames or IP addresses for <code>node-b</code> and <code>node-c</code>.</p>
1109 <div class="paragraph">
1110 <p>Even if you did want to run a RegionServer on <code>node-a</code>, you should refer to it by the hostname the other servers would use to communicate with it.
1111 In this case, that would be <code>node-a.example.com</code>.
1112 This enables you to distribute the configuration to each node of your cluster any hostname conflicts.
1113 Save the file.</p>
1114 </div>
1115 </li>
1116 <li>
1117 <p>Configure HBase to use <code>node-b</code> as a backup master.</p>
1118 <div class="paragraph">
1119 <p>Create a new file in <em>conf/</em> called <em>backup-masters</em>, and add a new line to it with the hostname for <code>node-b</code>.
1120 In this demonstration, the hostname is <code>node-b.example.com</code>.</p>
1121 </div>
1122 </li>
1123 <li>
1124 <p>Configure ZooKeeper</p>
1125 <div class="paragraph">
1126 <p>In reality, you should carefully consider your ZooKeeper configuration.
1127 You can find out more about configuring ZooKeeper in <a href="#zookeeper">zookeeper</a> section.
1128 This configuration will direct HBase to start and manage a ZooKeeper instance on each node of the cluster.</p>
1129 </div>
1130 <div class="paragraph">
1131 <p>On <code>node-a</code>, edit <em>conf/hbase-site.xml</em> and add the following properties.</p>
1132 </div>
1133 <div class="listingblock">
1134 <div class="content">
1135 <pre class="CodeRay highlight"><code data-lang="xml"><span class="tag">&lt;property&gt;</span>
1136 <span class="tag">&lt;name&gt;</span>hbase.zookeeper.quorum<span class="tag">&lt;/name&gt;</span>
1137 <span class="tag">&lt;value&gt;</span>node-a.example.com,node-b.example.com,node-c.example.com<span class="tag">&lt;/value&gt;</span>
1138 <span class="tag">&lt;/property&gt;</span>
1139 <span class="tag">&lt;property&gt;</span>
1140 <span class="tag">&lt;name&gt;</span>hbase.zookeeper.property.dataDir<span class="tag">&lt;/name&gt;</span>
1141 <span class="tag">&lt;value&gt;</span>/usr/local/zookeeper<span class="tag">&lt;/value&gt;</span>
1142 <span class="tag">&lt;/property&gt;</span></code></pre>
1143 </div>
1144 </div>
1145 </li>
1146 <li>
1147 <p>Everywhere in your configuration that you have referred to <code>node-a</code> as <code>localhost</code>, change the reference to point to the hostname that the other nodes will use to refer to <code>node-a</code>.
1148 In these examples, the hostname is <code>node-a.example.com</code>.</p>
1149 </li>
1150 </ol>
1151 </div>
1152 <div class="paragraph">
1153 <div class="title">Procedure: Prepare <code>node-b</code> and <code>node-c</code></div>
1154 <p><code>node-b</code> will run a backup master server and a ZooKeeper instance.</p>
1155 </div>
1156 <div class="olist arabic">
1157 <ol class="arabic">
1158 <li>
1159 <p>Download and unpack HBase.</p>
1160 <div class="paragraph">
1161 <p>Download and unpack HBase to <code>node-b</code>, just as you did for the standalone and pseudo-distributed quickstarts.</p>
1162 </div>
1163 </li>
1164 <li>
1165 <p>Copy the configuration files from <code>node-a</code> to <code>node-b</code>.and <code>node-c</code>.</p>
1166 <div class="paragraph">
1167 <p>Each node of your cluster needs to have the same configuration information.
1168 Copy the contents of the <em>conf/</em> directory to the <em>conf/</em> directory on <code>node-b</code> and <code>node-c</code>.</p>
1169 </div>
1170 </li>
1171 </ol>
1172 </div>
1173 <div class="olist arabic">
1174 <div class="title">Procedure: Start and Test Your Cluster</div>
1175 <ol class="arabic">
1176 <li>
1177 <p>Be sure HBase is not running on any node.</p>
1178 <div class="paragraph">
1179 <p>If you forgot to stop HBase from previous testing, you will have errors.
1180 Check to see whether HBase is running on any of your nodes by using the <code>jps</code> command.
1181 Look for the processes <code>HMaster</code>, <code>HRegionServer</code>, and <code>HQuorumPeer</code>.
1182 If they exist, kill them.</p>
1183 </div>
1184 </li>
1185 <li>
1186 <p>Start the cluster.</p>
1187 <div class="paragraph">
1188 <p>On <code>node-a</code>, issue the <code>start-hbase.sh</code> command.
1189 Your output will be similar to that below.</p>
1190 </div>
1191 <div class="listingblock">
1192 <div class="content">
1193 <pre>$ bin/start-hbase.sh
1194 node-c.example.com: starting zookeeper, logging to /home/hbuser/hbase-0.98.3-hadoop2/bin/../logs/hbase-hbuser-zookeeper-node-c.example.com.out
1195 node-a.example.com: starting zookeeper, logging to /home/hbuser/hbase-0.98.3-hadoop2/bin/../logs/hbase-hbuser-zookeeper-node-a.example.com.out
1196 node-b.example.com: starting zookeeper, logging to /home/hbuser/hbase-0.98.3-hadoop2/bin/../logs/hbase-hbuser-zookeeper-node-b.example.com.out
1197 starting master, logging to /home/hbuser/hbase-0.98.3-hadoop2/bin/../logs/hbase-hbuser-master-node-a.example.com.out
1198 node-c.example.com: starting regionserver, logging to /home/hbuser/hbase-0.98.3-hadoop2/bin/../logs/hbase-hbuser-regionserver-node-c.example.com.out
1199 node-b.example.com: starting regionserver, logging to /home/hbuser/hbase-0.98.3-hadoop2/bin/../logs/hbase-hbuser-regionserver-node-b.example.com.out
1200 node-b.example.com: starting master, logging to /home/hbuser/hbase-0.98.3-hadoop2/bin/../logs/hbase-hbuser-master-nodeb.example.com.out</pre>
1201 </div>
1202 </div>
1203 <div class="paragraph">
1204 <p>ZooKeeper starts first, followed by the master, then the RegionServers, and finally the backup masters.</p>
1205 </div>
1206 </li>
1207 <li>
1208 <p>Verify that the processes are running.</p>
1209 <div class="paragraph">
1210 <p>On each node of the cluster, run the <code>jps</code> command and verify that the correct processes are running on each server.
1211 You may see additional Java processes running on your servers as well, if they are used for other purposes.</p>
1212 </div>
1213 <div class="exampleblock">
1214 <div class="title">Example 2. <code>node-a</code> <code>jps</code> Output</div>
1215 <div class="content">
1216 <div class="listingblock">
1217 <div class="content">
1218 <pre>$ jps
1219 20355 Jps
1220 20071 HQuorumPeer
1221 20137 HMaster</pre>
1222 </div>
1223 </div>
1224 </div>
1225 </div>
1226 <div class="exampleblock">
1227 <div class="title">Example 3. <code>node-b</code> <code>jps</code> Output</div>
1228 <div class="content">
1229 <div class="listingblock">
1230 <div class="content">
1231 <pre>$ jps
1232 15930 HRegionServer
1233 16194 Jps
1234 15838 HQuorumPeer
1235 16010 HMaster</pre>
1236 </div>
1237 </div>
1238 </div>
1239 </div>
1240 <div class="exampleblock">
1241 <div class="title">Example 4. <code>node-c</code> <code>jps</code> Output</div>
1242 <div class="content">
1243 <div class="listingblock">
1244 <div class="content">
1245 <pre>$ jps
1246 13901 Jps
1247 13639 HQuorumPeer
1248 13737 HRegionServer</pre>
1249 </div>
1250 </div>
1251 </div>
1252 </div>
1253 <div class="admonitionblock note">
1254 <table>
1255 <tr>
1256 <td class="icon">
1257 <i class="fa icon-note" title="Note"></i>
1258 </td>
1259 <td class="content">
1260 <div class="title">ZooKeeper Process Name</div>
1261 <div class="paragraph">
1262 <p>The <code>HQuorumPeer</code> process is a ZooKeeper instance which is controlled and started by HBase.
1263 If you use ZooKeeper this way, it is limited to one instance per cluster node and is appropriate for testing only.
1264 If ZooKeeper is run outside of HBase, the process is called <code>QuorumPeer</code>.
1265 For more about ZooKeeper configuration, including using an external ZooKeeper instance with HBase, see <a href="#zookeeper">zookeeper</a> section.</p>
1266 </div>
1267 </td>
1268 </tr>
1269 </table>
1270 </div>
1271 </li>
1272 <li>
1273 <p>Browse to the Web UI.</p>
1274 <div class="admonitionblock note">
1275 <table>
1276 <tr>
1277 <td class="icon">
1278 <i class="fa icon-note" title="Note"></i>
1279 </td>
1280 <td class="content">
1281 <div class="title">Web UI Port Changes</div>
1282 Web UI Port Changes
1283 </td>
1284 </tr>
1285 </table>
1286 </div>
1287 <div class="paragraph">
1288 <p>In HBase newer than 0.98.x, the HTTP ports used by the HBase Web UI changed from 60010 for the
1289 Master and 60030 for each RegionServer to 16010 for the Master and 16030 for the RegionServer.</p>
1290 </div>
1291 <div class="paragraph">
1292 <p>If everything is set up correctly, you should be able to connect to the UI for the Master
1293 <code><a href="http://node-a.example.com:16010/" class="bare">http://node-a.example.com:16010/</a></code> or the secondary master at <code><a href="http://node-b.example.com:16010/" class="bare">http://node-b.example.com:16010/</a></code>
1294 using a web browser.
1295 If you can connect via <code>localhost</code> but not from another host, check your firewall rules.
1296 You can see the web UI for each of the RegionServers at port 16030 of their IP addresses, or by
1297 clicking their links in the web UI for the Master.</p>
1298 </div>
1299 </li>
1300 <li>
1301 <p>Test what happens when nodes or services disappear.</p>
1302 <div class="paragraph">
1303 <p>With a three-node cluster you have configured, things will not be very resilient.
1304 You can still test the behavior of the primary Master or a RegionServer by killing the associated processes and watching the logs.</p>
1305 </div>
1306 </li>
1307 </ol>
1308 </div>
1309 </div>
1310 <div class="sect2">
1311 <h3 id="_where_to_go_next"><a class="anchor" href="#_where_to_go_next"></a>2.5. Where to go next</h3>
1312 <div class="paragraph">
1313 <p>The next chapter, <a href="#configuration">configuration</a>, gives more information about the different HBase run modes, system requirements for running HBase, and critical configuration areas for setting up a distributed HBase cluster.</p>
1314 </div>
1315 </div>
1316 </div>
1317 </div>
1318 <h1 id="configuration" class="sect0"><a class="anchor" href="#configuration"></a>Apache HBase Configuration</h1>
1319 <div class="openblock partintro">
1320 <div class="content">
1321 This chapter expands upon the <a href="#getting_started">Getting Started</a> chapter to further explain configuration of Apache HBase.
1322 Please read this chapter carefully, especially the <a href="#basic.prerequisites">Basic Prerequisites</a>
1323 to ensure that your HBase testing and deployment goes smoothly, and prevent data loss.
1324 Familiarize yourself with <a href="#hbase_supported_tested_definitions">Support and Testing Expectations</a> as well.
1325 </div>
1326 </div>
1327 <div class="sect1">
1328 <h2 id="_configuration_files"><a class="anchor" href="#_configuration_files"></a>3. Configuration Files</h2>
1329 <div class="sectionbody">
1330 <div class="paragraph">
1331 <p>Apache HBase uses the same configuration system as Apache Hadoop.
1332 All configuration files are located in the <em>conf/</em> directory, which needs to be kept in sync for each node on your cluster.</p>
1333 </div>
1334 <div class="dlist">
1335 <div class="title">HBase Configuration File Descriptions</div>
1336 <dl>
1337 <dt class="hdlist1"><em>backup-masters</em></dt>
1338 <dd>
1339 <p>Not present by default.
1340 A plain-text file which lists hosts on which the Master should start a backup Master process, one host per line.</p>
1341 </dd>
1342 <dt class="hdlist1"><em>hadoop-metrics2-hbase.properties</em></dt>
1343 <dd>
1344 <p>Used to connect HBase Hadoop&#8217;s Metrics2 framework.
1345 See the <a href="https://wiki.apache.org/hadoop/HADOOP-6728-MetricsV2">Hadoop Wiki entry</a> for more information on Metrics2.
1346 Contains only commented-out examples by default.</p>
1347 </dd>
1348 <dt class="hdlist1"><em>hbase-env.cmd</em> and <em>hbase-env.sh</em></dt>
1349 <dd>
1350 <p>Script for Windows and Linux / Unix environments to set up the working environment for HBase, including the location of Java, Java options, and other environment variables.
1351 The file contains many commented-out examples to provide guidance.</p>
1352 </dd>
1353 <dt class="hdlist1"><em>hbase-policy.xml</em></dt>
1354 <dd>
1355 <p>The default policy configuration file used by RPC servers to make authorization decisions on client requests.
1356 Only used if HBase <a href="#security">security</a> is enabled.</p>
1357 </dd>
1358 <dt class="hdlist1"><em>hbase-site.xml</em></dt>
1359 <dd>
1360 <p>The main HBase configuration file.
1361 This file specifies configuration options which override HBase&#8217;s default configuration.
1362 You can view (but do not edit) the default configuration file at <em>docs/hbase-default.xml</em>.
1363 You can also view the entire effective configuration for your cluster (defaults and overrides) in the <span class="label">HBase Configuration</span> tab of the HBase Web UI.</p>
1364 </dd>
1365 <dt class="hdlist1"><em>log4j.properties</em></dt>
1366 <dd>
1367 <p>Configuration file for HBase logging via <code>log4j</code>.</p>
1368 </dd>
1369 <dt class="hdlist1"><em>regionservers</em></dt>
1370 <dd>
1371 <p>A plain-text file containing a list of hosts which should run a RegionServer in your HBase cluster.
1372 By default this file contains the single entry <code>localhost</code>.
1373 It should contain a list of hostnames or IP addresses, one per line, and should only contain <code>localhost</code> if each node in your cluster will run a RegionServer on its <code>localhost</code> interface.</p>
1374 </dd>
1375 </dl>
1376 </div>
1377 <div class="admonitionblock tip">
1378 <table>
1379 <tr>
1380 <td class="icon">
1381 <i class="fa icon-tip" title="Tip"></i>
1382 </td>
1383 <td class="content">
1384 <div class="title">Checking XML Validity</div>
1385 <div class="paragraph">
1386 <p>When you edit XML, it is a good idea to use an XML-aware editor to be sure that your syntax is correct and your XML is well-formed.
1387 You can also use the <code>xmllint</code> utility to check that your XML is well-formed.
1388 By default, <code>xmllint</code> re-flows and prints the XML to standard output.
1389 To check for well-formedness and only print output if errors exist, use the command <code>xmllint -noout filename.xml</code>.</p>
1390 </div>
1391 </td>
1392 </tr>
1393 </table>
1394 </div>
1395 <div class="admonitionblock warning">
1396 <table>
1397 <tr>
1398 <td class="icon">
1399 <i class="fa icon-warning" title="Warning"></i>
1400 </td>
1401 <td class="content">
1402 <div class="title">Keep Configuration In Sync Across the Cluster</div>
1403 <div class="paragraph">
1404 <p>When running in distributed mode, after you make an edit to an HBase configuration, make sure you copy the contents of the <em>conf/</em> directory to all nodes of the cluster.
1405 HBase will not do this for you.
1406 Use <code>rsync</code>, <code>scp</code>, or another secure mechanism for copying the configuration files to your nodes.
1407 For most configurations, a restart is needed for servers to pick up changes. Dynamic configuration is an exception to this, to be described later below.</p>
1408 </div>
1409 </td>
1410 </tr>
1411 </table>
1412 </div>
1413 </div>
1414 </div>
1415 <div class="sect1">
1416 <h2 id="basic.prerequisites"><a class="anchor" href="#basic.prerequisites"></a>4. Basic Prerequisites</h2>
1417 <div class="sectionbody">
1418 <div class="paragraph">
1419 <p>This section lists required services and some required system configuration.</p>
1420 </div>
1421 <table id="java" class="tableblock frame-all grid-all spread">
1422 <caption class="title">Table 2. Java</caption>
1423 <colgroup>
1424 <col style="width: 16.6666%;">
1425 <col style="width: 16.6666%;">
1426 <col style="width: 66.6668%;">
1427 </colgroup>
1428 <thead>
1429 <tr>
1430 <th class="tableblock halign-left valign-top">HBase Version</th>
1431 <th class="tableblock halign-left valign-top">JDK 7</th>
1432 <th class="tableblock halign-left valign-top">JDK 8</th>
1433 </tr>
1434 </thead>
1435 <tbody>
1436 <tr>
1437 <td class="tableblock halign-left valign-top"><p class="tableblock">2.0</p></td>
1438 <td class="tableblock halign-left valign-top"><p class="tableblock"><a href="http://search-hadoop.com/m/YGbbsPxZ723m3as">Not Supported</a></p></td>
1439 <td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
1440 </tr>
1441 <tr>
1442 <td class="tableblock halign-left valign-top"><p class="tableblock">1.3</p></td>
1443 <td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
1444 <td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
1445 </tr>
1446 <tr>
1447 <td class="tableblock halign-left valign-top"><p class="tableblock">1.2</p></td>
1448 <td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
1449 <td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
1450 </tr>
1451 </tbody>
1452 </table>
1453 <div class="admonitionblock note">
1454 <table>
1455 <tr>
1456 <td class="icon">
1457 <i class="fa icon-note" title="Note"></i>
1458 </td>
1459 <td class="content">
1460 HBase will neither build nor compile with Java 6.
1461 </td>
1462 </tr>
1463 </table>
1464 </div>
1465 <div class="admonitionblock note">
1466 <table>
1467 <tr>
1468 <td class="icon">
1469 <i class="fa icon-note" title="Note"></i>
1470 </td>
1471 <td class="content">
1472 You must set <code>JAVA_HOME</code> on each node of your cluster. <em>hbase-env.sh</em> provides a handy mechanism to do this.
1473 </td>
1474 </tr>
1475 </table>
1476 </div>
1477 <div id="os" class="dlist">
1478 <div class="title">Operating System Utilities</div>
1479 <dl>
1480 <dt class="hdlist1">ssh</dt>
1481 <dd>
1482 <p>HBase uses the Secure Shell (ssh) command and utilities extensively to communicate between cluster nodes. Each server in the cluster must be running <code>ssh</code> so that the Hadoop and HBase daemons can be managed. You must be able to connect to all nodes via SSH, including the local node, from the Master as well as any backup Master, using a shared key rather than a password. You can see the basic methodology for such a set-up in Linux or Unix systems at "<a href="#passwordless.ssh.quickstart">Procedure: Configure Passwordless SSH Access</a>". If your cluster nodes use OS X, see the section, <a href="https://wiki.apache.org/hadoop/Running_Hadoop_On_OS_X_10.5_64-bit_%28Single-Node_Cluster%29">SSH: Setting up Remote Desktop and Enabling Self-Login</a> on the Hadoop wiki.</p>
1483 </dd>
1484 <dt class="hdlist1">DNS</dt>
1485 <dd>
1486 <p>HBase uses the local hostname to self-report its IP address. Both forward and reverse DNS resolving must work in versions of HBase previous to 0.92.0. The <a href="https://github.com/sujee/hadoop-dns-checker">hadoop-dns-checker</a> tool can be used to verify DNS is working correctly on the cluster. The project <code>README</code> file provides detailed instructions on usage.</p>
1487 </dd>
1488 <dt class="hdlist1">Loopback IP</dt>
1489 <dd>
1490 <p>Prior to hbase-0.96.0, HBase only used the IP address <code>127.0.0.1</code> to refer to <code>localhost</code>, and this was not configurable.
1491 See <a href="#loopback.ip">Loopback IP</a> for more details.</p>
1492 </dd>
1493 <dt class="hdlist1">NTP</dt>
1494 <dd>
1495 <p>The clocks on cluster nodes should be synchronized. A small amount of variation is acceptable, but larger amounts of skew can cause erratic and unexpected behavior. Time synchronization is one of the first things to check if you see unexplained problems in your cluster. It is recommended that you run a Network Time Protocol (NTP) service, or another time-synchronization mechanism on your cluster and that all nodes look to the same service for time synchronization. See the <a href="http://www.tldp.org/LDP/sag/html/basic-ntp-config.html">Basic NTP Configuration</a> at <em class="citetitle">The Linux Documentation Project (TLDP)</em> to set up NTP.</p>
1496 </dd>
1497 </dl>
1498 </div>
1499 <div id="ulimit" class="dlist">
1500 <dl>
1501 <dt class="hdlist1">Limits on Number of Files and Processes (ulimit)</dt>
1502 <dd>
1503 <p>Apache HBase is a database. It requires the ability to open a large number of files at once. Many Linux distributions limit the number of files a single user is allowed to open to <code>1024</code> (or <code>256</code> on older versions of OS X). You can check this limit on your servers by running the command <code>ulimit -n</code> when logged in as the user which runs HBase. See <a href="#trouble.rs.runtime.filehandles">the Troubleshooting section</a> for some of the problems you may experience if the limit is too low. You may also notice errors such as the following:</p>
1504 <div class="listingblock">
1505 <div class="content">
1506 <pre>2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Exception increateBlockOutputStream java.io.EOFException
1507 2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-6935524980745310745_1391901</pre>
1508 </div>
1509 </div>
1510 <div class="paragraph">
1511 <p>It is recommended to raise the ulimit to at least 10,000, but more likely 10,240, because the value is usually expressed in multiples of 1024. Each ColumnFamily has at least one StoreFile, and possibly more than six StoreFiles if the region is under load. The number of open files required depends upon the number of ColumnFamilies and the number of regions. The following is a rough formula for calculating the potential number of open files on a RegionServer.</p>
1512 </div>
1513 <div class="listingblock">
1514 <div class="title">Calculate the Potential Number of Open Files</div>
1515 <div class="content">
1516 <pre>(StoreFiles per ColumnFamily) x (regions per RegionServer)</pre>
1517 </div>
1518 </div>
1519 <div class="paragraph">
1520 <p>For example, assuming that a schema had 3 ColumnFamilies per region with an average of 3 StoreFiles per ColumnFamily, and there are 100 regions per RegionServer, the JVM will open <code>3 * 3 * 100 = 900</code> file descriptors, not counting open JAR files, configuration files, and others. Opening a file does not take many resources, and the risk of allowing a user to open too many files is minimal.</p>
1521 </div>
1522 <div class="paragraph">
1523 <p>Another related setting is the number of processes a user is allowed to run at once. In Linux and Unix, the number of processes is set using the <code>ulimit -u</code> command. This should not be confused with the <code>nproc</code> command, which controls the number of CPUs available to a given user. Under load, a <code>ulimit -u</code> that is too low can cause OutOfMemoryError exceptions. See Jack Levin&#8217;s major HDFS issues thread on the hbase-users mailing list, from 2011.</p>
1524 </div>
1525 <div class="paragraph">
1526 <p>Configuring the maximum number of file descriptors and processes for the user who is running the HBase process is an operating system configuration, rather than an HBase configuration. It is also important to be sure that the settings are changed for the user that actually runs HBase. To see which user started HBase, and that user&#8217;s ulimit configuration, look at the first line of the HBase log for that instance. A useful read setting config on your hadoop cluster is Aaron Kimball&#8217;s Configuration Parameters: What can you just ignore?</p>
1527 </div>
1528 <div class="exampleblock">
1529 <div class="title">Example 5. <code>ulimit</code> Settings on Ubuntu</div>
1530 <div class="content">
1531 <div class="paragraph">
1532 <p>To configure ulimit settings on Ubuntu, edit <em>/etc/security/limits.conf</em>, which is a space-delimited file with four columns. Refer to the man page for <em>limits.conf</em> for details about the format of this file. In the following example, the first line sets both soft and hard limits for the number of open files (nofile) to 32768 for the operating system user with the username hadoop. The second line sets the number of processes to 32000 for the same user.</p>
1533 </div>
1534 <div class="listingblock">
1535 <div class="content">
1536 <pre>hadoop - nofile 32768
1537 hadoop - nproc 32000</pre>
1538 </div>
1539 </div>
1540 <div class="paragraph">
1541 <p>The settings are only applied if the Pluggable Authentication Module (PAM) environment is directed to use them. To configure PAM to use these limits, be sure that the <em>/etc/pam.d/common-session</em> file contains the following line:</p>
1542 </div>
1543 <div class="listingblock">
1544 <div class="content">
1545 <pre>session required pam_limits.so</pre>
1546 </div>
1547 </div>
1548 </div>
1549 </div>
1550 </dd>
1551 <dt class="hdlist1">Linux Shell</dt>
1552 <dd>
1553 <p>All of the shell scripts that come with HBase rely on the <a href="http://www.gnu.org/software/bash">GNU Bash</a> shell.</p>
1554 </dd>
1555 <dt class="hdlist1">Windows</dt>
1556 <dd>
1557 <p>Prior to HBase 0.96, running HBase on Microsoft Windows was limited only for testing purposes.
1558 Running production systems on Windows machines is not recommended.</p>
1559 </dd>
1560 </dl>
1561 </div>
1562 <div class="sect2">
1563 <h3 id="hadoop"><a class="anchor" href="#hadoop"></a>4.1. <a href="https://hadoop.apache.org">Hadoop</a></h3>
1564 <div class="paragraph">
1565 <p>The following table summarizes the versions of Hadoop supported with each version of HBase.
1566 Based on the version of HBase, you should select the most appropriate version of Hadoop.
1567 You can use Apache Hadoop, or a vendor&#8217;s distribution of Hadoop.
1568 No distinction is made here.
1569 See <a href="https://wiki.apache.org/hadoop/Distributions%20and%20Commercial%20Support">the Hadoop wiki</a> for information about vendors of Hadoop.</p>
1570 </div>
1571 <div class="admonitionblock tip">
1572 <table>
1573 <tr>
1574 <td class="icon">
1575 <i class="fa icon-tip" title="Tip"></i>
1576 </td>
1577 <td class="content">
1578 <div class="title">Hadoop 2.x is recommended.</div>
1579 <div class="paragraph">
1580 <p>Hadoop 2.x is faster and includes features, such as short-circuit reads, which will help improve your HBase random read profile.
1581 Hadoop 2.x also includes important bug fixes that will improve your overall HBase experience. HBase does not support running with
1582 earlier versions of Hadoop. See the table below for requirements specific to different HBase versions.</p>
1583 </div>
1584 <div class="paragraph">
1585 <p>Hadoop 3.x is still in early access releases and has not yet been sufficiently tested by the HBase community for production use cases.</p>
1586 </div>
1587 </td>
1588 </tr>
1589 </table>
1590 </div>
1591 <div class="paragraph">
1592 <p>Use the following legend to interpret this table:</p>
1593 </div>
1594 <div class="ulist">
1595 <div class="title">Hadoop version support matrix</div>
1596 <ul>
1597 <li>
1598 <p>"S" = supported</p>
1599 </li>
1600 <li>
1601 <p>"X" = not supported</p>
1602 </li>
1603 <li>
1604 <p>"NT" = Not tested</p>
1605 </li>
1606 </ul>
1607 </div>
1608 <table class="tableblock frame-all grid-all spread">
1609 <colgroup>
1610 <col style="width: 25%;">
1611 <col style="width: 25%;">
1612 <col style="width: 25%;">
1613 <col style="width: 25%;">
1614 </colgroup>
1615 <thead>
1616 <tr>
1617 <th class="tableblock halign-left valign-top"></th>
1618 <th class="tableblock halign-left valign-top">HBase-1.2.x</th>
1619 <th class="tableblock halign-left valign-top">HBase-1.3.x</th>
1620 <th class="tableblock halign-left valign-top">HBase-2.0.x</th>
1621 </tr>
1622 </thead>
1623 <tbody>
1624 <tr>
1625 <td class="tableblock halign-left valign-top"><p class="tableblock">Hadoop-2.0.x-alpha</p></td>
1626 <td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td>
1627 <td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td>
1628 <td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td>
1629 </tr>
1630 <tr>
1631 <td class="tableblock halign-left valign-top"><p class="tableblock">Hadoop-2.1.0-beta</p></td>
1632 <td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td>
1633 <td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td>
1634 <td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td>
1635 </tr>
1636 <tr>
1637 <td class="tableblock halign-left valign-top"><p class="tableblock">Hadoop-2.2.0</p></td>
1638 <td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td>
1639 <td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td>
1640 <td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td>
1641 </tr>
1642 <tr>
1643 <td class="tableblock halign-left valign-top"><p class="tableblock">Hadoop-2.3.x</p></td>
1644 <td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td>
1645 <td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td>
1646 <td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td>
1647 </tr>
1648 <tr>
1649 <td class="tableblock halign-left valign-top"><p class="tableblock">Hadoop-2.4.x</p></td>
1650 <td class="tableblock halign-left valign-top"><p class="tableblock">S</p></td>
1651 <td class="tableblock halign-left valign-top"><p class="tableblock">S</p></td>
1652 <td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td>
1653 </tr>
1654 <tr>
1655 <td class="tableblock halign-left valign-top"><p class="tableblock">Hadoop-2.5.x</p></td>
1656 <td class="tableblock halign-left valign-top"><p class="tableblock">S</p></td>
1657 <td class="tableblock halign-left valign-top"><p class="tableblock">S</p></td>
1658 <td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td>
1659 </tr>
1660 <tr>
1661 <td class="tableblock halign-left valign-top"><p class="tableblock">Hadoop-2.6.0</p></td>
1662 <td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td>
1663 <td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td>
1664 <td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td>
1665 </tr>
1666 <tr>
1667 <td class="tableblock halign-left valign-top"><p class="tableblock">Hadoop-2.6.1+</p></td>
1668 <td class="tableblock halign-left valign-top"><p class="tableblock">S</p></td>
1669 <td class="tableblock halign-left valign-top"><p class="tableblock">S</p></td>
1670 <td class="tableblock halign-left valign-top"><p class="tableblock">S</p></td>
1671 </tr>
1672 <tr>
1673 <td class="tableblock halign-left valign-top"><p class="tableblock">Hadoop-2.7.0</p></td>
1674 <td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td>
1675 <td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td>
1676 <td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td>
1677 </tr>
1678 <tr>
1679 <td class="tableblock halign-left valign-top"><p class="tableblock">Hadoop-2.7.1+</p></td>
1680 <td class="tableblock halign-left valign-top"><p class="tableblock">S</p></td>
1681 <td class="tableblock halign-left valign-top"><p class="tableblock">S</p></td>
1682 <td class="tableblock halign-left valign-top"><p class="tableblock">S</p></td>
1683 </tr>
1684 <tr>
1685 <td class="tableblock halign-left valign-top"><p class="tableblock">Hadoop-2.8.0</p></td>
1686 <td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td>
1687 <td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td>
1688 <td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td>
1689 </tr>
1690 <tr>
1691 <td class="tableblock halign-left valign-top"><p class="tableblock">Hadoop-2.8.1</p></td>
1692 <td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td>
1693 <td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td>
1694 <td class="tableblock halign-left valign-top"><p class="tableblock">X</p></td>
1695 </tr>
1696 <tr>
1697 <td class="tableblock halign-left valign-top"><p class="tableblock">Hadoop-3.0.0</p></td>
1698 <td class="tableblock halign-left valign-top"><p class="tableblock">NT</p></td>
1699 <td class="tableblock halign-left valign-top"><p class="tableblock">NT</p></td>
1700 <td class="tableblock halign-left valign-top"><p class="tableblock">NT</p></td>
1701 </tr>
1702 </tbody>
1703 </table>
1704 <div class="admonitionblock tip">
1705 <table>
1706 <tr>
1707 <td class="icon">
1708 <i class="fa icon-tip" title="Tip"></i>
1709 </td>
1710 <td class="content">
1711 <div class="title">Hadoop Pre-2.6.1 and JDK 1.8 Kerberos</div>
1712 <div class="paragraph">
1713 <p>When using pre-2.6.1 Hadoop versions and JDK 1.8 in a Kerberos environment, HBase server can fail
1714 and abort due to Kerberos keytab relogin error. Late version of JDK 1.7 (1.7.0_80) has the problem too.
1715 Refer to <a href="https://issues.apache.org/jira/browse/HADOOP-10786">HADOOP-10786</a> for additional details.
1716 Consider upgrading to Hadoop 2.6.1+ in this case.</p>
1717 </div>
1718 </td>
1719 </tr>
1720 </table>
1721 </div>
1722 <div class="admonitionblock tip">
1723 <table>
1724 <tr>
1725 <td class="icon">
1726 <i class="fa icon-tip" title="Tip"></i>
1727 </td>
1728 <td class="content">
1729 <div class="title">Hadoop 2.6.x</div>
1730 <div class="paragraph">
1731 <p>Hadoop distributions based on the 2.6.x line <strong>must</strong> have
1732 <a href="https://issues.apache.org/jira/browse/HADOOP-11710">HADOOP-11710</a> applied if you plan to run
1733 HBase on top of an HDFS Encryption Zone. Failure to do so will result in cluster failure and
1734 data loss. This patch is present in Apache Hadoop releases 2.6.1+.</p>
1735 </div>
1736 </td>
1737 </tr>
1738 </table>
1739 </div>
1740 <div class="admonitionblock tip">
1741 <table>
1742 <tr>
1743 <td class="icon">
1744 <i class="fa icon-tip" title="Tip"></i>
1745 </td>
1746 <td class="content">
1747 <div class="title">Hadoop 2.7.x</div>
1748 <div class="paragraph">
1749 <p>Hadoop version 2.7.0 is not tested or supported as the Hadoop PMC has explicitly labeled that release as not being stable. (reference the <a href="https://s.apache.org/hadoop-2.7.0-announcement">announcement of Apache Hadoop 2.7.0</a>.)</p>
1750 </div>
1751 </td>
1752 </tr>
1753 </table>
1754 </div>
1755 <div class="admonitionblock tip">
1756 <table>
1757 <tr>
1758 <td class="icon">
1759 <i class="fa icon-tip" title="Tip"></i>
1760 </td>
1761 <td class="content">
1762 <div class="title">Hadoop 2.8.x</div>
1763 <div class="paragraph">
1764 <p>Hadoop version 2.8.0 and 2.8.1 are not tested or supported as the Hadoop PMC has explicitly labeled that releases as not being stable. (reference the <a href="https://s.apache.org/hadoop-2.8.0-announcement">announcement of Apache Hadoop 2.8.0</a> and <a href="https://s.apache.org/hadoop-2.8.1-announcement">announcement of Apache Hadoop 2.8.1</a>.)</p>
1765 </div>
1766 </td>
1767 </tr>
1768 </table>
1769 </div>
1770 <div class="admonitionblock note">
1771 <table>
1772 <tr>
1773 <td class="icon">
1774 <i class="fa icon-note" title="Note"></i>
1775 </td>
1776 <td class="content">
1777 <div class="title">Replace the Hadoop Bundled With HBase!</div>
1778 <div class="paragraph">
1779 <p>Because HBase depends on Hadoop, it bundles an instance of the Hadoop jar under its <em>lib</em> directory.
1780 The bundled jar is ONLY for use in standalone mode.
1781 In distributed mode, it is <em>critical</em> that the version of Hadoop that is out on your cluster match what is under HBase.
1782 Replace the hadoop jar found in the HBase lib directory with the hadoop jar you are running on your cluster to avoid version mismatch issues.
1783 Make sure you replace the jar in HBase across your whole cluster.
1784 Hadoop version mismatch issues have various manifestations but often all look like its hung.</p>
1785 </div>
1786 </td>
1787 </tr>
1788 </table>
1789 </div>
1790 <div class="sect3">
1791 <h4 id="dfs.datanode.max.transfer.threads"><a class="anchor" href="#dfs.datanode.max.transfer.threads"></a>4.1.1. <code>dfs.datanode.max.transfer.threads</code> </h4>
1792 <div class="paragraph">
1793 <p>An HDFS DataNode has an upper bound on the number of files that it will serve at any one time.
1794 Before doing any loading, make sure you have configured Hadoop&#8217;s <em>conf/hdfs-site.xml</em>, setting the <code>dfs.datanode.max.transfer.threads</code> value to at least the following:</p>
1795 </div>
1796 <div class="listingblock">
1797 <div class="content">
1798 <pre class="CodeRay highlight"><code data-lang="xml"><span class="tag">&lt;property&gt;</span>
1799 <span class="tag">&lt;name&gt;</span>dfs.datanode.max.transfer.threads<span class="tag">&lt;/name&gt;</span>
1800 <span class="tag">&lt;value&gt;</span>4096<span class="tag">&lt;/value&gt;</span>
1801 <span class="tag">&lt;/property&gt;</span></code></pre>
1802 </div>
1803 </div>
1804 <div class="paragraph">
1805 <p>Be sure to restart your HDFS after making the above configuration.</p>
1806 </div>
1807 <div class="paragraph">
1808 <p>Not having this configuration in place makes for strange-looking failures.
1809 One manifestation is a complaint about missing blocks.
1810 For example:</p>
1811 </div>
1812 <div class="listingblock">
1813 <div class="content">
1814 <pre>10/12/08 20:10:31 INFO hdfs.DFSClient: Could not obtain block
1815 blk_XXXXXXXXXXXXXXXXXXXXXX_YYYYYYYY from any node: java.io.IOException: No live nodes
1816 contain current block. Will get new block locations from namenode and retry...</pre>
1817 </div>
1818 </div>
1819 <div class="paragraph">
1820 <p>See also <a href="#casestudies.max.transfer.threads">casestudies.max.transfer.threads</a> and note that this property was previously known as <code>dfs.datanode.max.xcievers</code> (e.g. <a href="http://ccgtech.blogspot.com/2010/02/hadoop-hdfs-deceived-by-xciever.html">Hadoop HDFS: Deceived by Xciever</a>).</p>
1821 </div>
1822 </div>
1823 </div>
1824 <div class="sect2">
1825 <h3 id="zookeeper.requirements"><a class="anchor" href="#zookeeper.requirements"></a>4.2. ZooKeeper Requirements</h3>
1826 <div class="paragraph">
1827 <p>ZooKeeper 3.4.x is required.
1828 HBase makes use of the <code>multi</code> functionality that is only available since Zookeeper 3.4.0. The <code>hbase.zookeeper.useMulti</code> configuration property defaults to <code>true</code>.
1829 Refer to <a href="https://issues.apache.org/jira/browse/HBASE-12241">HBASE-12241 (The crash of regionServer when taking deadserver&#8217;s replication queue breaks replication)</a> and <a href="https://issues.apache.org/jira/browse/HBASE-6775">HBASE-6775 (Use ZK.multi when available for HBASE-6710 0.92/0.94 compatibility fix)</a> for background.
1830 The property is deprecated and useMulti is always enabled in HBase 2.0.</p>
1831 </div>
1832 </div>
1833 </div>
1834 </div>
1835 <div class="sect1">
1836 <h2 id="standalone_dist"><a class="anchor" href="#standalone_dist"></a>5. HBase run modes: Standalone and Distributed</h2>
1837 <div class="sectionbody">
1838 <div class="paragraph">
1839 <p>HBase has two run modes: <a href="#standalone">standalone</a> and <a href="#distributed">distributed</a>.
1840 Out of the box, HBase runs in standalone mode.
1841 Whatever your mode, you will need to configure HBase by editing files in the HBase <em>conf</em> directory.
1842 At a minimum, you must edit <span class="code">conf/hbase-env.sh</span> to tell HBase which java to use.
1843 In this file you set HBase environment variables such as the heapsize and other options for the <code>JVM</code>, the preferred location for log files, etc.
1844 Set <span class="var">JAVA_HOME</span> to point at the root of your java install.</p>
1845 </div>
1846 <div class="sect2">
1847 <h3 id="standalone"><a class="anchor" href="#standalone"></a>5.1. Standalone HBase</h3>
1848 <div class="paragraph">
1849 <p>This is the default mode.
1850 Standalone mode is what is described in the <a href="#quickstart">quickstart</a> section.
1851 In standalone mode, HBase does not use HDFS&#8201;&#8212;&#8201;it uses the local filesystem instead&#8201;&#8212;&#8201;and it runs all HBase daemons and a local ZooKeeper all up in the same JVM.
1852 ZooKeeper binds to a well known port so clients may talk to HBase.</p>
1853 </div>
1854 <div class="sect3">
1855 <h4 id="standalone.over.hdfs"><a class="anchor" href="#standalone.over.hdfs"></a>5.1.1. Standalone HBase over HDFS</h4>
1856 <div class="paragraph">
1857 <p>A sometimes useful variation on standalone hbase has all daemons running inside the
1858 one JVM but rather than persist to the local filesystem, instead
1859 they persist to an HDFS instance.</p>
1860 </div>
1861 <div class="paragraph">
1862 <p>You might consider this profile when you are intent on
1863 a simple deploy profile, the loading is light, but the
1864 data must persist across node comings and goings. Writing to
1865 HDFS where data is replicated ensures the latter.</p>
1866 </div>
1867 <div class="paragraph">
1868 <p>To configure this standalone variant, edit your <em>hbase-site.xml</em>
1869 setting <em>hbase.rootdir</em> to point at a directory in your
1870 HDFS instance but then set <em>hbase.cluster.distributed</em>
1871 to <em>false</em>. For example:</p>
1872 </div>
1873 <div class="listingblock">
1874 <div class="content">
1875 <pre class="CodeRay highlight"><code data-lang="xml"><span class="tag">&lt;configuration&gt;</span>
1876 <span class="tag">&lt;property&gt;</span>
1877 <span class="tag">&lt;name&gt;</span>hbase.rootdir<span class="tag">&lt;/name&gt;</span>
1878 <span class="tag">&lt;value&gt;</span>hdfs://namenode.example.org:8020/hbase<span class="tag">&lt;/value&gt;</span>
1879 <span class="tag">&lt;/property&gt;</span>
1880 <span class="tag">&lt;property&gt;</span>
1881 <span class="tag">&lt;name&gt;</span>hbase.cluster.distributed<span class="tag">&lt;/name&gt;</span>
1882 <span class="tag">&lt;value&gt;</span>false<span class="tag">&lt;/value&gt;</span>
1883 <span class="tag">&lt;/property&gt;</span>
1884 <span class="tag">&lt;/configuration&gt;</span></code></pre>
1885 </div>
1886 </div>
1887 </div>
1888 </div>
1889 <div class="sect2">
1890 <h3 id="distributed"><a class="anchor" href="#distributed"></a>5.2. Distributed</h3>
1891 <div class="paragraph">
1892 <p>Distributed mode can be subdivided into distributed but all daemons run on a single node&#8201;&#8212;&#8201;a.k.a. <em>pseudo-distributed</em>&#8201;&#8212;&#8201;and <em>fully-distributed</em> where the daemons are spread across all nodes in the cluster.
1893 The <em>pseudo-distributed</em> vs. <em>fully-distributed</em> nomenclature comes from Hadoop.</p>
1894 </div>
1895 <div class="paragraph">
1896 <p>Pseudo-distributed mode can run against the local filesystem or it can run against an instance of the <em>Hadoop Distributed File System</em> (HDFS). Fully-distributed mode can ONLY run on HDFS.
1897 See the Hadoop <a href="https://hadoop.apache.org/docs/current/">documentation</a> for how to set up HDFS.
1898 A good walk-through for setting up HDFS on Hadoop 2 can be found at <a href="http://www.alexjf.net/blog/distributed-systems/hadoop-yarn-installation-definitive-guide" class="bare">http://www.alexjf.net/blog/distributed-systems/hadoop-yarn-installation-definitive-guide</a>.</p>
1899 </div>
1900 <div class="sect3">
1901 <h4 id="pseudo"><a class="anchor" href="#pseudo"></a>5.2.1. Pseudo-distributed</h4>
1902 <div class="admonitionblock note">
1903 <table>
1904 <tr>
1905 <td class="icon">
1906 <i class="fa icon-note" title="Note"></i>
1907 </td>
1908 <td class="content">
1909 <div class="title">Pseudo-Distributed Quickstart</div>
1910 <div class="paragraph">
1911 <p>A quickstart has been added to the <a href="#quickstart">quickstart</a> chapter.
1912 See <a href="#quickstart_pseudo">quickstart-pseudo</a>.
1913 Some of the information that was originally in this section has been moved there.</p>
1914 </div>
1915 </td>
1916 </tr>
1917 </table>
1918 </div>
1919 <div class="paragraph">
1920 <p>A pseudo-distributed mode is simply a fully-distributed mode run on a single host.
1921 Use this HBase configuration for testing and prototyping purposes only.
1922 Do not use this configuration for production or for performance evaluation.</p>
1923 </div>
1924 </div>
1925 </div>
1926 <div class="sect2">
1927 <h3 id="fully_dist"><a class="anchor" href="#fully_dist"></a>5.3. Fully-distributed</h3>
1928 <div class="paragraph">
1929 <p>By default, HBase runs in standalone mode.
1930 Both standalone mode and pseudo-distributed mode are provided for the purposes of small-scale testing.
1931 For a production environment, distributed mode is advised.
1932 In distributed mode, multiple instances of HBase daemons run on multiple servers in the cluster.</p>
1933 </div>
1934 <div class="paragraph">
1935 <p>Just as in pseudo-distributed mode, a fully distributed configuration requires that you set the <code>hbase.cluster.distributed</code> property to <code>true</code>.
1936 Typically, the <code>hbase.rootdir</code> is configured to point to a highly-available HDFS filesystem.</p>
1937 </div>
1938 <div class="paragraph">
1939 <p>In addition, the cluster is configured so that multiple cluster nodes enlist as RegionServers, ZooKeeper QuorumPeers, and backup HMaster servers.
1940 These configuration basics are all demonstrated in <a href="#quickstart_fully_distributed">quickstart-fully-distributed</a>.</p>
1941 </div>
1942 <div class="paragraph">
1943 <div class="title">Distributed RegionServers</div>
1944 <p>Typically, your cluster will contain multiple RegionServers all running on different servers, as well as primary and backup Master and ZooKeeper daemons.
1945 The <em>conf/regionservers</em> file on the master server contains a list of hosts whose RegionServers are associated with this cluster.
1946 Each host is on a separate line.
1947 All hosts listed in this file will have their RegionServer processes started and stopped when the master server starts or stops.</p>
1948 </div>
1949 <div class="paragraph">
1950 <div class="title">ZooKeeper and HBase</div>
1951 <p>See the <a href="#zookeeper">ZooKeeper</a> section for ZooKeeper setup instructions for HBase.</p>
1952 </div>
1953 <div class="exampleblock">
1954 <div class="title">Example 6. Example Distributed HBase Cluster</div>
1955 <div class="content">
1956 <div class="paragraph">
1957 <p>This is a bare-bones <em>conf/hbase-site.xml</em> for a distributed HBase cluster.
1958 A cluster that is used for real-world work would contain more custom configuration parameters.
1959 Most HBase configuration directives have default values, which are used unless the value is overridden in the <em>hbase-site.xml</em>.
1960 See "<a href="#config.files">Configuration Files</a>" for more information.</p>
1961 </div>
1962 <div class="listingblock">
1963 <div class="content">
1964 <pre class="CodeRay highlight"><code data-lang="xml"><span class="tag">&lt;configuration&gt;</span>
1965 <span class="tag">&lt;property&gt;</span>
1966 <span class="tag">&lt;name&gt;</span>hbase.rootdir<span class="tag">&lt;/name&gt;</span>
1967 <span class="tag">&lt;value&gt;</span>hdfs://namenode.example.org:8020/hbase<span class="tag">&lt;/value&gt;</span>
1968 <span class="tag">&lt;/property&gt;</span>
1969 <span class="tag">&lt;property&gt;</span>
1970 <span class="tag">&lt;name&gt;</span>hbase.cluster.distributed<span class="tag">&lt;/name&gt;</span>
1971 <span class="tag">&lt;value&gt;</span>true<span class="tag">&lt;/value&gt;</span>
1972 <span class="tag">&lt;/property&gt;</span>
1973 <span class="tag">&lt;property&gt;</span>
1974 <span class="tag">&lt;name&gt;</span>hbase.zookeeper.quorum<span class="tag">&lt;/name&gt;</span>
1975 <span class="tag">&lt;value&gt;</span>node-a.example.com,node-b.example.com,node-c.example.com<span class="tag">&lt;/value&gt;</span>
1976 <span class="tag">&lt;/property&gt;</span>
1977 <span class="tag">&lt;/configuration&gt;</span></code></pre>
1978 </div>
1979 </div>
1980 <div class="paragraph">
1981 <p>This is an example <em>conf/regionservers</em> file, which contains a list of nodes that should run a RegionServer in the cluster.
1982 These nodes need HBase installed and they need to use the same contents of the <em>conf/</em> directory as the Master server</p>
1983 </div>
1984 <div class="listingblock">
1985 <div class="content">
1986 <pre class="CodeRay highlight"><code data-lang="java">node-a.example.com
1987 node-b.example.com
1988 node-c.example.com</code></pre>
1989 </div>
1990 </div>
1991 <div class="paragraph">
1992 <p>This is an example <em>conf/backup-masters</em> file, which contains a list of each node that should run a backup Master instance.
1993 The backup Master instances will sit idle unless the main Master becomes unavailable.</p>
1994 </div>
1995 <div class="listingblock">
1996 <div class="content">
1997 <pre class="CodeRay highlight"><code data-lang="java">node-b.example.com
1998 node-c.example.com</code></pre>
1999 </div>
2000 </div>
2001 </div>
2002 </div>
2003 <div class="paragraph">
2004 <div class="title">Distributed HBase Quickstart</div>
2005 <p>See <a href="#quickstart_fully_distributed">quickstart-fully-distributed</a> for a walk-through of a simple three-node cluster configuration with multiple ZooKeeper, backup HMaster, and RegionServer instances.</p>
2006 </div>
2007 <div class="olist arabic">
2008 <div class="title">Procedure: HDFS Client Configuration</div>
2009 <ol class="arabic">
2010 <li>
2011 <p>Of note, if you have made HDFS client configuration changes on your Hadoop cluster, such as configuration directives for HDFS clients, as opposed to server-side configurations, you must use one of the following methods to enable HBase to see and use these configuration changes:</p>
2012 <div class="olist loweralpha">
2013 <ol class="loweralpha" type="a">
2014 <li>
2015 <p>Add a pointer to your <code>HADOOP_CONF_DIR</code> to the <code>HBASE_CLASSPATH</code> environment variable in <em>hbase-env.sh</em>.</p>
2016 </li>
2017 <li>
2018 <p>Add a copy of <em>hdfs-site.xml</em> (or <em>hadoop-site.xml</em>) or, better, symlinks, under <em>${HBASE_HOME}/conf</em>, or</p>
2019 </li>
2020 <li>
2021 <p>if only a small set of HDFS client configurations, add them to <em>hbase-site.xml</em>.</p>
2022 </li>
2023 </ol>
2024 </div>
2025 </li>
2026 </ol>
2027 </div>
2028 <div class="paragraph">
2029 <p>An example of such an HDFS client configuration is <code>dfs.replication</code>.
2030 If for example, you want to run with a replication factor of 5, HBase will create files with the default of 3 unless you do the above to make the configuration available to HBase.</p>
2031 </div>
2032 </div>
2033 </div>
2034 </div>
2035 <div class="sect1">
2036 <h2 id="confirm"><a class="anchor" href="#confirm"></a>6. Running and Confirming Your Installation</h2>
2037 <div class="sectionbody">
2038 <div class="paragraph">
2039 <p>Make sure HDFS is running first.
2040 Start and stop the Hadoop HDFS daemons by running <em>bin/start-hdfs.sh</em> over in the <code>HADOOP_HOME</code> directory.
2041 You can ensure it started properly by testing the <code>put</code> and <code>get</code> of files into the Hadoop filesystem.
2042 HBase does not normally use the MapReduce or YARN daemons. These do not need to be started.</p>
2043 </div>
2044 <div class="paragraph">
2045 <p><em>If</em> you are managing your own ZooKeeper, start it and confirm it&#8217;s running, else HBase will start up ZooKeeper for you as part of its start process.</p>
2046 </div>
2047 <div class="paragraph">
2048 <p>Start HBase with the following command:</p>
2049 </div>
2050 <div class="listingblock">
2051 <div class="content">
2052 <pre>bin/start-hbase.sh</pre>
2053 </div>
2054 </div>
2055 <div class="paragraph">
2056 <p>Run the above from the <code>HBASE_HOME</code> directory.</p>
2057 </div>
2058 <div class="paragraph">
2059 <p>You should now have a running HBase instance.
2060 HBase logs can be found in the <em>logs</em> subdirectory.
2061 Check them out especially if HBase had trouble starting.</p>
2062 </div>
2063 <div class="paragraph">
2064 <p>HBase also puts up a UI listing vital attributes.
2065 By default it&#8217;s deployed on the Master host at port 16010 (HBase RegionServers listen on port 16020 by default and put up an informational HTTP server at port 16030). If the Master is running on a host named <code>master.example.org</code> on the default port, point your browser at http://master.example.org:16010 to see the web interface.</p>
2066 </div>
2067 <div class="paragraph">
2068 <p>Once HBase has started, see the <a href="#shell_exercises">shell exercises</a> section for how to create tables, add data, scan your insertions, and finally disable and drop your tables.</p>
2069 </div>
2070 <div class="paragraph">
2071 <p>To stop HBase after exiting the HBase shell enter</p>
2072 </div>
2073 <div class="listingblock">
2074 <div class="content">
2075 <pre>$ ./bin/stop-hbase.sh
2076 stopping hbase...............</pre>
2077 </div>
2078 </div>
2079 <div class="paragraph">
2080 <p>Shutdown can take a moment to complete.
2081 It can take longer if your cluster is comprised of many machines.
2082 If you are running a distributed operation, be sure to wait until HBase has shut down completely before stopping the Hadoop daemons.</p>
2083 </div>
2084 </div>
2085 </div>
2086 <div class="sect1">
2087 <h2 id="config.files"><a class="anchor" href="#config.files"></a>7. Default Configuration</h2>
2088 <div class="sectionbody">
2089 <div class="sect2">
2090 <h3 id="hbase.site"><a class="anchor" href="#hbase.site"></a>7.1. <em>hbase-site.xml</em> and <em>hbase-default.xml</em></h3>
2091 <div class="paragraph">
2092 <p>Just as in Hadoop where you add site-specific HDFS configuration to the <em>hdfs-site.xml</em> file, for HBase, site specific customizations go into the file <em>conf/hbase-site.xml</em>.
2093 For the list of configurable properties, see <a href="#hbase_default_configurations">hbase default configurations</a> below or view the raw <em>hbase-default.xml</em> source file in the HBase source code at <em>src/main/resources</em>.</p>
2094 </div>
2095 <div class="paragraph">
2096 <p>Not all configuration options make it out to <em>hbase-default.xml</em>.
2097 Some configurations would only appear in source code; the only way to identify these changes are through code review.</p>
2098 </div>
2099 <div class="paragraph">
2100 <p>Currently, changes here will require a cluster restart for HBase to notice the change.</p>
2101 </div>
2102 </div>
2103 <div class="sect2">
2104 <h3 id="hbase_default_configurations"><a class="anchor" href="#hbase_default_configurations"></a>7.2. HBase Default Configuration</h3>
2105 <div class="paragraph">
2106 <p>The documentation below is generated using the default hbase configuration file, <em>hbase-default.xml</em>, as source.</p>
2107 </div>
2108 <div id="hbase.tmp.dir" class="dlist">
2109 <dl>
2110 <dt class="hdlist1"><code>hbase.tmp.dir</code></dt>
2111 <dd>
2112 <div class="paragraph">
2113 <div class="title">Description</div>
2114 <p>Temporary directory on the local filesystem. Change this setting to point to a location more permanent than '/tmp', the usual resolve for java.io.tmpdir, as the '/tmp' directory is cleared on machine restart.</p>
2115 </div>
2116 <div class="paragraph">
2117 <div class="title">Default</div>
2118 <p><code>${java.io.tmpdir}/hbase-${user.name}</code></p>
2119 </div>
2120 </dd>
2121 </dl>
2122 </div>
2123 <div id="hbase.rootdir" class="dlist">
2124 <dl>
2125 <dt class="hdlist1"><code>hbase.rootdir</code></dt>
2126 <dd>
2127 <div class="paragraph">
2128 <div class="title">Description</div>
2129 <p>The directory shared by region servers and into which HBase persists. The URL should be 'fully-qualified' to include the filesystem scheme. For example, to specify the HDFS directory '/hbase' where the HDFS instance&#8217;s namenode is running at namenode.example.org on port 9000, set this value to: hdfs://namenode.example.org:9000/hbase. By default, we write to whatever ${hbase.tmp.dir} is set too&#8201;&#8212;&#8201;usually /tmp&#8201;&#8212;&#8201;so change this configuration or else all data will be lost on machine restart.</p>
2130 </div>
2131 <div class="paragraph">
2132 <div class="title">Default</div>
2133 <p><code>${hbase.tmp.dir}/hbase</code></p>
2134 </div>
2135 </dd>
2136 </dl>
2137 </div>
2138 <div id="hbase.cluster.distributed" class="dlist">
2139 <dl>
2140 <dt class="hdlist1"><code>hbase.cluster.distributed</code></dt>
2141 <dd>
2142 <div class="paragraph">
2143 <div class="title">Description</div>
2144 <p>The mode the cluster will be in. Possible values are false for standalone mode and true for distributed mode. If false, startup will run all HBase and ZooKeeper daemons together in the one JVM.</p>
2145 </div>
2146 <div class="paragraph">
2147 <div class="title">Default</div>
2148 <p><code>false</code></p>
2149 </div>
2150 </dd>
2151 </dl>
2152 </div>
2153 <div id="hbase.zookeeper.quorum" class="dlist">
2154 <dl>
2155 <dt class="hdlist1"><code>hbase.zookeeper.quorum</code></dt>
2156 <dd>
2157 <div class="paragraph">
2158 <div class="title">Description</div>
2159 <p>Comma separated list of servers in the ZooKeeper ensemble (This config. should have been named hbase.zookeeper.ensemble). For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com". By default this is set to localhost for local and pseudo-distributed modes of operation. For a fully-distributed setup, this should be set to a full list of ZooKeeper ensemble servers. If HBASE_MANAGES_ZK is set in hbase-env.sh this is the list of servers which hbase will start/stop ZooKeeper on as part of cluster start/stop. Client-side, we will take this list of ensemble members and put it together with the hbase.zookeeper.property.clientPort config. and pass it into zookeeper constructor as the connectString parameter.</p>
2160 </div>
2161 <div class="paragraph">
2162 <div class="title">Default</div>
2163 <p><code>localhost</code></p>
2164 </div>
2165 </dd>
2166 </dl>
2167 </div>
2168 <div id="zookeeper.recovery.retry.maxsleeptime" class="dlist">
2169 <dl>
2170 <dt class="hdlist1"><code>zookeeper.recovery.retry.maxsleeptime</code></dt>
2171 <dd>
2172 <div class="paragraph">
2173 <div class="title">Description</div>
2174 <p>Max sleep time before retry zookeeper operations in milliseconds, a max time is needed here so that sleep time won&#8217;t grow unboundedly</p>
2175 </div>
2176 <div class="paragraph">
2177 <div class="title">Default</div>
2178 <p><code>60000</code></p>
2179 </div>
2180 </dd>
2181 </dl>
2182 </div>
2183 <div id="hbase.local.dir" class="dlist">
2184 <dl>
2185 <dt class="hdlist1"><code>hbase.local.dir</code></dt>
2186 <dd>
2187 <div class="paragraph">
2188 <div class="title">Description</div>
2189 <p>Directory on the local filesystem to be used as a local storage.</p>
2190 </div>
2191 <div class="paragraph">
2192 <div class="title">Default</div>
2193 <p><code>${hbase.tmp.dir}/local/</code></p>
2194 </div>
2195 </dd>
2196 </dl>
2197 </div>
2198 <div id="hbase.master.port" class="dlist">
2199 <dl>
2200 <dt class="hdlist1"><code>hbase.master.port</code></dt>
2201 <dd>
2202 <div class="paragraph">
2203 <div class="title">Description</div>
2204 <p>The port the HBase Master should bind to.</p>
2205 </div>
2206 <div class="paragraph">
2207 <div class="title">Default</div>
2208 <p><code>16000</code></p>
2209 </div>
2210 </dd>
2211 </dl>
2212 </div>
2213 <div id="hbase.master.info.port" class="dlist">
2214 <dl>
2215 <dt class="hdlist1"><code>hbase.master.info.port</code></dt>
2216 <dd>
2217 <div class="paragraph">
2218 <div class="title">Description</div>
2219 <p>The port for the HBase Master web UI. Set to -1 if you do not want a UI instance run.</p>
2220 </div>
2221 <div class="paragraph">
2222 <div class="title">Default</div>
2223 <p><code>16010</code></p>
2224 </div>
2225 </dd>
2226 </dl>
2227 </div>
2228 <div id="hbase.master.info.bindAddress" class="dlist">
2229 <dl>
2230 <dt class="hdlist1"><code>hbase.master.info.bindAddress</code></dt>
2231 <dd>
2232 <div class="paragraph">
2233 <div class="title">Description</div>
2234 <p>The bind address for the HBase Master web UI</p>
2235 </div>
2236 <div class="paragraph">
2237 <div class="title">Default</div>
2238 <p><code>0.0.0.0</code></p>
2239 </div>
2240 </dd>
2241 </dl>
2242 </div>
2243 <div id="hbase.master.logcleaner.plugins" class="dlist">
2244 <dl>
2245 <dt class="hdlist1"><code>hbase.master.logcleaner.plugins</code></dt>
2246 <dd>
2247 <div class="paragraph">
2248 <div class="title">Description</div>
2249 <p>A comma-separated list of BaseLogCleanerDelegate invoked by the LogsCleaner service. These WAL cleaners are called in order, so put the cleaner that prunes the most files in front. To implement your own BaseLogCleanerDelegate, just put it in HBase&#8217;s classpath and add the fully qualified class name here. Always add the above default log cleaners in the list.</p>
2250 </div>
2251 <div class="paragraph">
2252 <div class="title">Default</div>
2253 <p><code>org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner,org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner</code></p>
2254 </div>
2255 </dd>
2256 </dl>
2257 </div>
2258 <div id="hbase.master.logcleaner.ttl" class="dlist">
2259 <dl>
2260 <dt class="hdlist1"><code>hbase.master.logcleaner.ttl</code></dt>
2261 <dd>
2262 <div class="paragraph">
2263 <div class="title">Description</div>
2264 <p>How long a WAL remain in the archive ({hbase.rootdir}/oldWALs) directory, after which it will be cleaned by a Master thread. The value is in milliseconds.</p>
2265 </div>
2266 <div class="paragraph">
2267 <div class="title">Default</div>
2268 <p><code>600000</code></p>
2269 </div>
2270 </dd>
2271 </dl>
2272 </div>
2273 <div id="hbase.master.procedurewalcleaner.ttl" class="dlist">
2274 <dl>
2275 <dt class="hdlist1"><code>hbase.master.procedurewalcleaner.ttl</code></dt>
2276 <dd>
2277 <div class="paragraph">
2278 <div class="title">Description</div>
2279 <p>How long a Procedure WAL will remain in the archive directory, after which it will be cleaned by a Master thread. The value is in milliseconds.</p>
2280 </div>
2281 <div class="paragraph">
2282 <div class="title">Default</div>
2283 <p><code>604800000</code></p>
2284 </div>
2285 </dd>
2286 </dl>
2287 </div>
2288 <div id="hbase.master.hfilecleaner.plugins" class="dlist">
2289 <dl>
2290 <dt class="hdlist1"><code>hbase.master.hfilecleaner.plugins</code></dt>
2291 <dd>
2292 <div class="paragraph">
2293 <div class="title">Description</div>
2294 <p>A comma-separated list of BaseHFileCleanerDelegate invoked by the HFileCleaner service. These HFiles cleaners are called in order, so put the cleaner that prunes the most files in front. To implement your own BaseHFileCleanerDelegate, just put it in HBase&#8217;s classpath and add the fully qualified class name here. Always add the above default log cleaners in the list as they will be overwritten in hbase-site.xml.</p>
2295 </div>
2296 <div class="paragraph">
2297 <div class="title">Default</div>
2298 <p><code>org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner</code></p>
2299 </div>
2300 </dd>
2301 </dl>
2302 </div>
2303 <div id="hbase.master.infoserver.redirect" class="dlist">
2304 <dl>
2305 <dt class="hdlist1"><code>hbase.master.infoserver.redirect</code></dt>
2306 <dd>
2307 <div class="paragraph">
2308 <div class="title">Description</div>
2309 <p>Whether or not the Master listens to the Master web UI port (hbase.master.info.port) and redirects requests to the web UI server shared by the Master and RegionServer. Config. makes sense when Master is serving Regions (not the default).</p>
2310 </div>
2311 <div class="paragraph">
2312 <div class="title">Default</div>
2313 <p><code>true</code></p>
2314 </div>
2315 </dd>
2316 </dl>
2317 </div>
2318 <div id="hbase.master.fileSplitTimeout" class="dlist">
2319 <dl>
2320 <dt class="hdlist1"><code>hbase.master.fileSplitTimeout</code></dt>
2321 <dd>
2322 <div class="paragraph">
2323 <div class="title">Description</div>
2324 <p>Splitting a region, how long to wait on the file-splitting step before aborting the attempt. Default: 600000. This setting used to be known as hbase.regionserver.fileSplitTimeout in hbase-1.x. Split is now run master-side hence the rename (If a 'hbase.master.fileSplitTimeout' setting found, will use it to prime the current 'hbase.master.fileSplitTimeout' Configuration.</p>
2325 </div>
2326 <div class="paragraph">
2327 <div class="title">Default</div>
2328 <p><code>600000</code></p>
2329 </div>
2330 </dd>
2331 </dl>
2332 </div>
2333 <div id="hbase.regionserver.port" class="dlist">
2334 <dl>
2335 <dt class="hdlist1"><code>hbase.regionserver.port</code></dt>
2336 <dd>
2337 <div class="paragraph">
2338 <div class="title">Description</div>
2339 <p>The port the HBase RegionServer binds to.</p>
2340 </div>
2341 <div class="paragraph">
2342 <div class="title">Default</div>
2343 <p><code>16020</code></p>
2344 </div>
2345 </dd>
2346 </dl>
2347 </div>
2348 <div id="hbase.regionserver.info.port" class="dlist">
2349 <dl>
2350 <dt class="hdlist1"><code>hbase.regionserver.info.port</code></dt>
2351 <dd>
2352 <div class="paragraph">
2353 <div class="title">Description</div>
2354 <p>The port for the HBase RegionServer web UI Set to -1 if you do not want the RegionServer UI to run.</p>
2355 </div>
2356 <div class="paragraph">
2357 <div class="title">Default</div>
2358 <p><code>16030</code></p>
2359 </div>
2360 </dd>
2361 </dl>
2362 </div>
2363 <div id="hbase.regionserver.info.bindAddress" class="dlist">
2364 <dl>
2365 <dt class="hdlist1"><code>hbase.regionserver.info.bindAddress</code></dt>
2366 <dd>
2367 <div class="paragraph">
2368 <div class="title">Description</div>
2369 <p>The address for the HBase RegionServer web UI</p>
2370 </div>
2371 <div class="paragraph">
2372 <div class="title">Default</div>
2373 <p><code>0.0.0.0</code></p>
2374 </div>
2375 </dd>
2376 </dl>
2377 </div>
2378 <div id="hbase.regionserver.info.port.auto" class="dlist">
2379 <dl>
2380 <dt class="hdlist1"><code>hbase.regionserver.info.port.auto</code></dt>
2381 <dd>
2382 <div class="paragraph">
2383 <div class="title">Description</div>
2384 <p>Whether or not the Master or RegionServer UI should search for a port to bind to. Enables automatic port search if hbase.regionserver.info.port is already in use. Useful for testing, turned off by default.</p>
2385 </div>
2386 <div class="paragraph">
2387 <div class="title">Default</div>
2388 <p><code>false</code></p>
2389 </div>
2390 </dd>
2391 </dl>
2392 </div>
2393 <div id="hbase.regionserver.handler.count" class="dlist">
2394 <dl>
2395 <dt class="hdlist1"><code>hbase.regionserver.handler.count</code></dt>
2396 <dd>
2397 <div class="paragraph">
2398 <div class="title">Description</div>
2399 <p>Count of RPC Listener instances spun up on RegionServers. Same property is used by the Master for count of master handlers. Too many handlers can be counter-productive. Make it a multiple of CPU count. If mostly read-only, handlers count close to cpu count does well. Start with twice the CPU count and tune from there.</p>
2400 </div>
2401 <div class="paragraph">
2402 <div class="title">Default</div>
2403 <p><code>30</code></p>
2404 </div>
2405 </dd>
2406 </dl>
2407 </div>
2408 <div id="hbase.ipc.server.callqueue.handler.factor" class="dlist">
2409 <dl>
2410 <dt class="hdlist1"><code>hbase.ipc.server.callqueue.handler.factor</code></dt>
2411 <dd>
2412 <div class="paragraph">
2413 <div class="title">Description</div>
2414 <p>Factor to determine the number of call queues. A value of 0 means a single queue shared between all the handlers. A value of 1 means that each handler has its own queue.</p>
2415 </div>
2416 <div class="paragraph">
2417 <div class="title">Default</div>
2418 <p><code>0.1</code></p>
2419 </div>
2420 </dd>
2421 </dl>
2422 </div>
2423 <div id="hbase.ipc.server.callqueue.read.ratio" class="dlist">
2424 <dl>
2425 <dt class="hdlist1"><code>hbase.ipc.server.callqueue.read.ratio</code></dt>
2426 <dd>
2427 <div class="paragraph">
2428 <div class="title">Description</div>
2429 <p>Split the call queues into read and write queues. The specified interval (which should be between 0.0 and 1.0) will be multiplied by the number of call queues. A value of 0 indicate to not split the call queues, meaning that both read and write requests will be pushed to the same set of queues. A value lower than 0.5 means that there will be less read queues than write queues. A value of 0.5 means there will be the same number of read and write queues. A value greater than 0.5 means that there will be more read queues than write queues. A value of 1.0 means that all the queues except one are used to dispatch read requests. Example: Given the total number of call queues being 10 a read.ratio of 0 means that: the 10 queues will contain both read/write requests. a read.ratio of 0.3 means that: 3 queues will contain only read requests and 7 queues will contain only write requests. a read.ratio of 0.5 means that: 5 queues will contain only read requests and 5 queues will contain only write requests. a read.ratio of 0.8 means that: 8 queues will contain only read requests and 2 queues will contain only write requests. a read.ratio of 1 means that: 9 queues will contain only read requests and 1 queues will contain only write requests.</p>
2430 </div>
2431 <div class="paragraph">
2432 <div class="title">Default</div>
2433 <p><code>0</code></p>
2434 </div>
2435 </dd>
2436 </dl>
2437 </div>
2438 <div id="hbase.ipc.server.callqueue.scan.ratio" class="dlist">
2439 <dl>
2440 <dt class="hdlist1"><code>hbase.ipc.server.callqueue.scan.ratio</code></dt>
2441 <dd>
2442 <div class="paragraph">
2443 <div class="title">Description</div>
2444 <p>Given the number of read call queues, calculated from the total number of call queues multiplied by the callqueue.read.ratio, the scan.ratio property will split the read call queues into small-read and long-read queues. A value lower than 0.5 means that there will be less long-read queues than short-read queues. A value of 0.5 means that there will be the same number of short-read and long-read queues. A value greater than 0.5 means that there will be more long-read queues than short-read queues A value of 0 or 1 indicate to use the same set of queues for gets and scans. Example: Given the total number of read call queues being 8 a scan.ratio of 0 or 1 means that: 8 queues will contain both long and short read requests. a scan.ratio of 0.3 means that: 2 queues will contain only long-read requests and 6 queues will contain only short-read requests. a scan.ratio of 0.5 means that: 4 queues will contain only long-read requests and 4 queues will contain only short-read requests. a scan.ratio of 0.8 means that: 6 queues will contain only long-read requests and 2 queues will contain only short-read requests.</p>
2445 </div>
2446 <div class="paragraph">
2447 <div class="title">Default</div>
2448 <p><code>0</code></p>
2449 </div>
2450 </dd>
2451 </dl>
2452 </div>
2453 <div id="hbase.regionserver.msginterval" class="dlist">
2454 <dl>
2455 <dt class="hdlist1"><code>hbase.regionserver.msginterval</code></dt>
2456 <dd>
2457 <div class="paragraph">
2458 <div class="title">Description</div>
2459 <p>Interval between messages from the RegionServer to Master in milliseconds.</p>
2460 </div>
2461 <div class="paragraph">
2462 <div class="title">Default</div>
2463 <p><code>3000</code></p>
2464 </div>
2465 </dd>
2466 </dl>
2467 </div>
2468 <div id="hbase.regionserver.logroll.period" class="dlist">
2469 <dl>
2470 <dt class="hdlist1"><code>hbase.regionserver.logroll.period</code></dt>
2471 <dd>
2472 <div class="paragraph">
2473 <div class="title">Description</div>
2474 <p>Period at which we will roll the commit log regardless of how many edits it has.</p>
2475 </div>
2476 <div class="paragraph">
2477 <div class="title">Default</div>
2478 <p><code>3600000</code></p>
2479 </div>
2480 </dd>
2481 </dl>
2482 </div>
2483 <div id="hbase.regionserver.logroll.errors.tolerated" class="dlist">
2484 <dl>
2485 <dt class="hdlist1"><code>hbase.regionserver.logroll.errors.tolerated</code></dt>
2486 <dd>
2487 <div class="paragraph">
2488 <div class="title">Description</div>
2489 <p>The number of consecutive WAL close errors we will allow before triggering a server abort. A setting of 0 will cause the region server to abort if closing the current WAL writer fails during log rolling. Even a small value (2 or 3) will allow a region server to ride over transient HDFS errors.</p>
2490 </div>
2491 <div class="paragraph">
2492 <div class="title">Default</div>
2493 <p><code>2</code></p>
2494 </div>
2495 </dd>
2496 </dl>
2497 </div>
2498 <div id="hbase.regionserver.hlog.reader.impl" class="dlist">
2499 <dl>
2500 <dt class="hdlist1"><code>hbase.regionserver.hlog.reader.impl</code></dt>
2501 <dd>
2502 <div class="paragraph">
2503 <div class="title">Description</div>
2504 <p>The WAL file reader implementation.</p>
2505 </div>
2506 <div class="paragraph">
2507 <div class="title">Default</div>
2508 <p><code>org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader</code></p>
2509 </div>
2510 </dd>
2511 </dl>
2512 </div>
2513 <div id="hbase.regionserver.hlog.writer.impl" class="dlist">
2514 <dl>
2515 <dt class="hdlist1"><code>hbase.regionserver.hlog.writer.impl</code></dt>
2516 <dd>
2517 <div class="paragraph">
2518 <div class="title">Description</div>
2519 <p>The WAL file writer implementation.</p>
2520 </div>
2521 <div class="paragraph">
2522 <div class="title">Default</div>
2523 <p><code>org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter</code></p>
2524 </div>
2525 </dd>
2526 </dl>
2527 </div>
2528 <div id="hbase.regionserver.global.memstore.size" class="dlist">
2529 <dl>
2530 <dt class="hdlist1"><code>hbase.regionserver.global.memstore.size</code></dt>
2531 <dd>
2532 <div class="paragraph">
2533 <div class="title">Description</div>
2534 <p>Maximum size of all memstores in a region server before new updates are blocked and flushes are forced. Defaults to 40% of heap (0.4). Updates are blocked and flushes are forced until size of all memstores in a region server hits hbase.regionserver.global.memstore.size.lower.limit. The default value in this configuration has been intentionally left empty in order to honor the old hbase.regionserver.global.memstore.upperLimit property if present.</p>
2535 </div>
2536 <div class="paragraph">
2537 <div class="title">Default</div>
2538 <p>none</p>
2539 </div>
2540 </dd>
2541 </dl>
2542 </div>
2543 <div id="hbase.regionserver.global.memstore.size.lower.limit" class="dlist">
2544 <dl>
2545 <dt class="hdlist1"><code>hbase.regionserver.global.memstore.size.lower.limit</code></dt>
2546 <dd>
2547 <div class="paragraph">
2548 <div class="title">Description</div>
2549 <p>Maximum size of all memstores in a region server before flushes are forced. Defaults to 95% of hbase.regionserver.global.memstore.size (0.95). A 100% value for this value causes the minimum possible flushing to occur when updates are blocked due to memstore limiting. The default value in this configuration has been intentionally left empty in order to honor the old hbase.regionserver.global.memstore.lowerLimit property if present.</p>
2550 </div>
2551 <div class="paragraph">
2552 <div class="title">Default</div>
2553 <p>none</p>
2554 </div>
2555 </dd>
2556 </dl>
2557 </div>
2558 <div id="hbase.systemtables.compacting.memstore.type" class="dlist">
2559 <dl>
2560 <dt class="hdlist1"><code>hbase.systemtables.compacting.memstore.type</code></dt>
2561 <dd>
2562 <div class="paragraph">
2563 <div class="title">Description</div>
2564 <p>Determines the type of memstore to be used for system tables like META, namespace tables etc. By default NONE is the type and hence we use the default memstore for all the system tables. If we need to use compacting memstore for system tables then set this property to BASIC/EAGER</p>
2565 </div>
2566 <div class="paragraph">
2567 <div class="title">Default</div>
2568 <p><code>NONE</code></p>
2569 </div>
2570 </dd>
2571 </dl>
2572 </div>
2573 <div id="hbase.regionserver.optionalcacheflushinterval" class="dlist">
2574 <dl>
2575 <dt class="hdlist1"><code>hbase.regionserver.optionalcacheflushinterval</code></dt>
2576 <dd>
2577 <div class="paragraph">
2578 <div class="title">Description</div>
2579 <p>Maximum amount of time an edit lives in memory before being automatically flushed. Default 1 hour. Set it to 0 to disable automatic flushing.</p>
2580 </div>
2581 <div class="paragraph">
2582 <div class="title">Default</div>
2583 <p><code>3600000</code></p>
2584 </div>
2585 </dd>
2586 </dl>
2587 </div>
2588 <div id="hbase.regionserver.dns.interface" class="dlist">
2589 <dl>
2590 <dt class="hdlist1"><code>hbase.regionserver.dns.interface</code></dt>
2591 <dd>
2592 <div class="paragraph">
2593 <div class="title">Description</div>
2594 <p>The name of the Network Interface from which a region server should report its IP address.</p>
2595 </div>
2596 <div class="paragraph">
2597 <div class="title">Default</div>
2598 <p><code>default</code></p>
2599 </div>
2600 </dd>
2601 </dl>
2602 </div>
2603 <div id="hbase.regionserver.dns.nameserver" class="dlist">
2604 <dl>
2605 <dt class="hdlist1"><code>hbase.regionserver.dns.nameserver</code></dt>
2606 <dd>
2607 <div class="paragraph">
2608 <div class="title">Description</div>
2609 <p>The host name or IP address of the name server (DNS) which a region server should use to determine the host name used by the master for communication and display purposes.</p>
2610 </div>
2611 <div class="paragraph">
2612 <div class="title">Default</div>
2613 <p><code>default</code></p>
2614 </div>
2615 </dd>
2616 </dl>
2617 </div>
2618 <div id="hbase.regionserver.region.split.policy" class="dlist">
2619 <dl>
2620 <dt class="hdlist1"><code>hbase.regionserver.region.split.policy</code></dt>
2621 <dd>
2622 <div class="paragraph">
2623 <div class="title">Description</div>
2624 <p>A split policy determines when a region should be split. The various other split policies that are available currently are BusyRegionSplitPolicy, ConstantSizeRegionSplitPolicy, DisabledRegionSplitPolicy, DelimitedKeyPrefixRegionSplitPolicy, KeyPrefixRegionSplitPolicy, and SteppingSplitPolicy. DisabledRegionSplitPolicy blocks manual region splitting.</p>
2625 </div>
2626 <div class="paragraph">
2627 <div class="title">Default</div>
2628 <p><code>org.apache.hadoop.hbase.regionserver.SteppingSplitPolicy</code></p>
2629 </div>
2630 </dd>
2631 </dl>
2632 </div>
2633 <div id="hbase.regionserver.regionSplitLimit" class="dlist">
2634 <dl>
2635 <dt class="hdlist1"><code>hbase.regionserver.regionSplitLimit</code></dt>
2636 <dd>
2637 <div class="paragraph">
2638 <div class="title">Description</div>
2639 <p>Limit for the number of regions after which no more region splitting should take place. This is not hard limit for the number of regions but acts as a guideline for the regionserver to stop splitting after a certain limit. Default is set to 1000.</p>
2640 </div>
2641 <div class="paragraph">
2642 <div class="title">Default</div>
2643 <p><code>1000</code></p>
2644 </div>
2645 </dd>
2646 </dl>
2647 </div>
2648 <div id="zookeeper.session.timeout" class="dlist">
2649 <dl>
2650 <dt class="hdlist1"><code>zookeeper.session.timeout</code></dt>
2651 <dd>
2652 <div class="paragraph">
2653 <div class="title">Description</div>
2654 <p>ZooKeeper session timeout in milliseconds. It is used in two different ways. First, this value is used in the ZK client that HBase uses to connect to the ensemble. It is also used by HBase when it starts a ZK server and it is passed as the 'maxSessionTimeout'. See <a href="http://hadoop.apache.org/zookeeper/docs/current/zookeeperProgrammers.html#ch_zkSessions" class="bare">http://hadoop.apache.org/zookeeper/docs/current/zookeeperProgrammers.html#ch_zkSessions</a>. For example, if an HBase region server connects to a ZK ensemble that&#8217;s also managed by HBase, then the session timeout will be the one specified by this configuration. But, a region server that connects to an ensemble managed with a different configuration will be subjected that ensemble&#8217;s maxSessionTimeout. So, even though HBase might propose using 90 seconds, the ensemble can have a max timeout lower than this and it will take precedence. The current default that ZK ships with is 40 seconds, which is lower than HBase&#8217;s.</p>
2655 </div>
2656 <div class="paragraph">
2657 <div class="title">Default</div>
2658 <p><code>90000</code></p>
2659 </div>
2660 </dd>
2661 </dl>
2662 </div>
2663 <div id="zookeeper.znode.parent" class="dlist">
2664 <dl>
2665 <dt class="hdlist1"><code>zookeeper.znode.parent</code></dt>
2666 <dd>
2667 <div class="paragraph">
2668 <div class="title">Description</div>
2669 <p>Root ZNode for HBase in ZooKeeper. All of HBase&#8217;s ZooKeeper files that are configured with a relative path will go under this node. By default, all of HBase&#8217;s ZooKeeper file paths are configured with a relative path, so they will all go under this directory unless changed.</p>
2670 </div>
2671 <div class="paragraph">
2672 <div class="title">Default</div>
2673 <p><code>/hbase</code></p>
2674 </div>
2675 </dd>
2676 </dl>
2677 </div>
2678 <div id="zookeeper.znode.acl.parent" class="dlist">
2679 <dl>
2680 <dt class="hdlist1"><code>zookeeper.znode.acl.parent</code></dt>
2681 <dd>
2682 <div class="paragraph">
2683 <div class="title">Description</div>
2684 <p>Root ZNode for access control lists.</p>
2685 </div>
2686 <div class="paragraph">
2687 <div class="title">Default</div>
2688 <p><code>acl</code></p>
2689 </div>
2690 </dd>
2691 </dl>
2692 </div>
2693 <div id="hbase.zookeeper.dns.interface" class="dlist">
2694 <dl>
2695 <dt class="hdlist1"><code>hbase.zookeeper.dns.interface</code></dt>
2696 <dd>
2697 <div class="paragraph">
2698 <div class="title">Description</div>
2699 <p>The name of the Network Interface from which a ZooKeeper server should report its IP address.</p>
2700 </div>
2701 <div class="paragraph">
2702 <div class="title">Default</div>
2703 <p><code>default</code></p>
2704 </div>
2705 </dd>
2706 </dl>
2707 </div>
2708 <div id="hbase.zookeeper.dns.nameserver" class="dlist">
2709 <dl>
2710 <dt class="hdlist1"><code>hbase.zookeeper.dns.nameserver</code></dt>
2711 <dd>
2712 <div class="paragraph">
2713 <div class="title">Description</div>
2714 <p>The host name or IP address of the name server (DNS) which a ZooKeeper server should use to determine the host name used by the master for communication and display purposes.</p>
2715 </div>
2716 <div class="paragraph">
2717 <div class="title">Default</div>
2718 <p><code>default</code></p>
2719 </div>
2720 </dd>
2721 </dl>
2722 </div>
2723 <div id="hbase.zookeeper.peerport" class="dlist">
2724 <dl>
2725 <dt class="hdlist1"><code>hbase.zookeeper.peerport</code></dt>
2726 <dd>
2727 <div class="paragraph">
2728 <div class="title">Description</div>
2729 <p>Port used by ZooKeeper peers to talk to each other. See <a href="http://hadoop.apache.org/zookeeper/docs/r3.1.1/zookeeperStarted.html#sc_RunningReplicatedZooKeeper" class="bare">http://hadoop.apache.org/zookeeper/docs/r3.1.1/zookeeperStarted.html#sc_RunningReplicatedZooKeeper</a> for more information.</p>
2730 </div>
2731 <div class="paragraph">
2732 <div class="title">Default</div>
2733 <p><code>2888</code></p>
2734 </div>
2735 </dd>
2736 </dl>
2737 </div>
2738 <div id="hbase.zookeeper.leaderport" class="dlist">
2739 <dl>
2740 <dt class="hdlist1"><code>hbase.zookeeper.leaderport</code></dt>
2741 <dd>
2742 <div class="paragraph">
2743 <div class="title">Description</div>
2744 <p>Port used by ZooKeeper for leader election. See <a href="http://hadoop.apache.org/zookeeper/docs/r3.1.1/zookeeperStarted.html#sc_RunningReplicatedZooKeeper" class="bare">http://hadoop.apache.org/zookeeper/docs/r3.1.1/zookeeperStarted.html#sc_RunningReplicatedZooKeeper</a> for more information.</p>
2745 </div>
2746 <div class="paragraph">
2747 <div class="title">Default</div>
2748 <p><code>3888</code></p>
2749 </div>
2750 </dd>
2751 </dl>
2752 </div>
2753 <div id="hbase.zookeeper.property.initLimit" class="dlist">
2754 <dl>
2755 <dt class="hdlist1"><code>hbase.zookeeper.property.initLimit</code></dt>
2756 <dd>
2757 <div class="paragraph">
2758 <div class="title">Description</div>
2759 <p>Property from ZooKeeper&#8217;s config zoo.cfg. The number of ticks that the initial synchronization phase can take.</p>
2760 </div>
2761 <div class="paragraph">
2762 <div class="title">Default</div>
2763 <p><code>10</code></p>
2764 </div>
2765 </dd>
2766 </dl>
2767 </div>
2768 <div id="hbase.zookeeper.property.syncLimit" class="dlist">
2769 <dl>
2770 <dt class="hdlist1"><code>hbase.zookeeper.property.syncLimit</code></dt>
2771 <dd>
2772 <div class="paragraph">
2773 <div class="title">Description</div>
2774 <p>Property from ZooKeeper&#8217;s config zoo.cfg. The number of ticks that can pass between sending a request and getting an acknowledgment.</p>
2775 </div>
2776 <div class="paragraph">
2777 <div class="title">Default</div>
2778 <p><code>5</code></p>
2779 </div>
2780 </dd>
2781 </dl>
2782 </div>
2783 <div id="hbase.zookeeper.property.dataDir" class="dlist">
2784 <dl>
2785 <dt class="hdlist1"><code>hbase.zookeeper.property.dataDir</code></dt>
2786 <dd>
2787 <div class="paragraph">
2788 <div class="title">Description</div>
2789 <p>Property from ZooKeeper&#8217;s config zoo.cfg. The directory where the snapshot is stored.</p>
2790 </div>
2791 <div class="paragraph">
2792 <div class="title">Default</div>
2793 <p><code>${hbase.tmp.dir}/zookeeper</code></p>
2794 </div>
2795 </dd>
2796 </dl>
2797 </div>
2798 <div id="hbase.zookeeper.property.clientPort" class="dlist">
2799 <dl>
2800 <dt class="hdlist1"><code>hbase.zookeeper.property.clientPort</code></dt>
2801 <dd>
2802 <div class="paragraph">
2803 <div class="title">Description</div>
2804 <p>Property from ZooKeeper&#8217;s config zoo.cfg. The port at which the clients will connect.</p>
2805 </div>
2806 <div class="paragraph">
2807 <div class="title">Default</div>
2808 <p><code>2181</code></p>
2809 </div>
2810 </dd>
2811 </dl>
2812 </div>
2813 <div id="hbase.zookeeper.property.maxClientCnxns" class="dlist">
2814 <dl>
2815 <dt class="hdlist1"><code>hbase.zookeeper.property.maxClientCnxns</code></dt>
2816 <dd>
2817 <div class="paragraph">
2818 <div class="title">Description</div>
2819 <p>Property from ZooKeeper&#8217;s config zoo.cfg. Limit on number of concurrent connections (at the socket level) that a single client, identified by IP address, may make to a single member of the ZooKeeper ensemble. Set high to avoid zk connection issues running standalone and pseudo-distributed.</p>
2820 </div>
2821 <div class="paragraph">
2822 <div class="title">Default</div>
2823 <p><code>300</code></p>
2824 </div>
2825 </dd>
2826 </dl>
2827 </div>
2828 <div id="hbase.client.write.buffer" class="dlist">
2829 <dl>
2830 <dt class="hdlist1"><code>hbase.client.write.buffer</code></dt>
2831 <dd>
2832 <div class="paragraph">
2833 <div class="title">Description</div>
2834 <p>Default size of the BufferedMutator write buffer in bytes. A bigger buffer takes more memory&#8201;&#8212;&#8201;on both the client and server side since server instantiates the passed write buffer to process it&#8201;&#8212;&#8201;but a larger buffer size reduces the number of RPCs made. For an estimate of server-side memory-used, evaluate hbase.client.write.buffer * hbase.regionserver.handler.count</p>
2835 </div>
2836 <div class="paragraph">
2837 <div class="title">Default</div>
2838 <p><code>2097152</code></p>
2839 </div>
2840 </dd>
2841 </dl>
2842 </div>
2843 <div id="hbase.client.pause" class="dlist">
2844 <dl>
2845 <dt class="hdlist1"><code>hbase.client.pause</code></dt>
2846 <dd>
2847 <div class="paragraph">
2848 <div class="title">Description</div>
2849 <p>General client pause value. Used mostly as value to wait before running a retry of a failed get, region lookup, etc. See hbase.client.retries.number for description of how we backoff from this initial pause amount and how this pause works w/ retries.</p>
2850 </div>
2851 <div class="paragraph">
2852 <div class="title">Default</div>
2853 <p><code>100</code></p>
2854 </div>
2855 </dd>
2856 </dl>
2857 </div>
2858 <div id="hbase.client.pause.cqtbe" class="dlist">
2859 <dl>
2860 <dt class="hdlist1"><code>hbase.client.pause.cqtbe</code></dt>
2861 <dd>
2862 <div class="paragraph">
2863 <div class="title">Description</div>
2864 <p>Whether or not to use a special client pause for CallQueueTooBigException (cqtbe). Set this property to a higher value than hbase.client.pause if you observe frequent CQTBE from the same RegionServer and the call queue there keeps full</p>
2865 </div>
2866 <div class="paragraph">
2867 <div class="title">Default</div>
2868 <p>none</p>
2869 </div>
2870 </dd>
2871 </dl>
2872 </div>
2873 <div id="hbase.client.retries.number" class="dlist">
2874 <dl>
2875 <dt class="hdlist1"><code>hbase.client.retries.number</code></dt>
2876 <dd>
2877 <div class="paragraph">
2878 <div class="title">Description</div>
2879 <p>Maximum retries. Used as maximum for all retryable operations such as the getting of a cell&#8217;s value, starting a row update, etc. Retry interval is a rough function based on hbase.client.pause. At first we retry at this interval but then with backoff, we pretty quickly reach retrying every ten seconds. See HConstants#RETRY_BACKOFF for how the backup ramps up. Change this setting and hbase.client.pause to suit your workload.</p>
2880 </div>
2881 <div class="paragraph">
2882 <div class="title">Default</div>
2883 <p><code>15</code></p>
2884 </div>
2885 </dd>
2886 </dl>
2887 </div>
2888 <div id="hbase.client.max.total.tasks" class="dlist">
2889 <dl>
2890 <dt class="hdlist1"><code>hbase.client.max.total.tasks</code></dt>
2891 <dd>
2892 <div class="paragraph">
2893 <div class="title">Description</div>
2894 <p>The maximum number of concurrent mutation tasks a single HTable instance will send to the cluster.</p>
2895 </div>
2896 <div class="paragraph">
2897 <div class="title">Default</div>
2898 <p><code>100</code></p>
2899 </div>
2900 </dd>
2901 </dl>
2902 </div>
2903 <div id="hbase.client.max.perserver.tasks" class="dlist">
2904 <dl>
2905 <dt class="hdlist1"><code>hbase.client.max.perserver.tasks</code></dt>
2906 <dd>
2907 <div class="paragraph">
2908 <div class="title">Description</div>
2909 <p>The maximum number of concurrent mutation tasks a single HTable instance will send to a single region server.</p>
2910 </div>
2911 <div class="paragraph">
2912 <div class="title">Default</div>
2913 <p><code>2</code></p>
2914 </div>
2915 </dd>
2916 </dl>
2917 </div>
2918 <div id="hbase.client.max.perregion.tasks" class="dlist">
2919 <dl>
2920 <dt class="hdlist1"><code>hbase.client.max.perregion.tasks</code></dt>
2921 <dd>
2922 <div class="paragraph">
2923 <div class="title">Description</div>
2924 <p>The maximum number of concurrent mutation tasks the client will maintain to a single Region. That is, if there is already hbase.client.max.perregion.tasks writes in progress for this region, new puts won&#8217;t be sent to this region until some writes finishes.</p>
2925 </div>
2926 <div class="paragraph">
2927 <div class="title">Default</div>
2928 <p><code>1</code></p>
2929 </div>
2930 </dd>
2931 </dl>
2932 </div>
2933 <div id="hbase.client.perserver.requests.threshold" class="dlist">
2934 <dl>
2935 <dt class="hdlist1"><code>hbase.client.perserver.requests.threshold</code></dt>
2936 <dd>
2937 <div class="paragraph">
2938 <div class="title">Description</div>
2939 <p>The max number of concurrent pending requests for one server in all client threads (process level). Exceeding requests will be thrown ServerTooBusyException immediately to prevent user&#8217;s threads being occupied and blocked by only one slow region server. If you use a fix number of threads to access HBase in a synchronous way, set this to a suitable value which is related to the number of threads will help you. See <a href="https://issues.apache.org/jira/browse/HBASE-16388" class="bare">https://issues.apache.org/jira/browse/HBASE-16388</a> for details.</p>
2940 </div>
2941 <div class="paragraph">
2942 <div class="title">Default</div>
2943 <p><code>2147483647</code></p>
2944 </div>
2945 </dd>
2946 </dl>
2947 </div>
2948 <div id="hbase.client.scanner.caching" class="dlist">
2949 <dl>
2950 <dt class="hdlist1"><code>hbase.client.scanner.caching</code></dt>
2951 <dd>
2952 <div class="paragraph">
2953 <div class="title">Description</div>
2954 <p>Number of rows that we try to fetch when calling next on a scanner if it is not served from (local, client) memory. This configuration works together with hbase.client.scanner.max.result.size to try and use the network efficiently. The default value is Integer.MAX_VALUE by default so that the network will fill the chunk size defined by hbase.client.scanner.max.result.size rather than be limited by a particular number of rows since the size of rows varies table to table. If you know ahead of time that you will not require more than a certain number of rows from a scan, this configuration should be set to that row limit via Scan#setCaching. Higher caching values will enable faster scanners but will eat up more memory and some calls of next may take longer and longer times when the cache is empty. Do not set this value such that the time between invocations is greater than the scanner timeout; i.e. hbase.client.scanner.timeout.period</p>
2955 </div>
2956 <div class="paragraph">
2957 <div class="title">Default</div>
2958 <p><code>2147483647</code></p>
2959 </div>
2960 </dd>
2961 </dl>
2962 </div>
2963 <div id="hbase.client.keyvalue.maxsize" class="dlist">
2964 <dl>
2965 <dt class="hdlist1"><code>hbase.client.keyvalue.maxsize</code></dt>
2966 <dd>
2967 <div class="paragraph">
2968 <div class="title">Description</div>
2969 <p>Specifies the combined maximum allowed size of a KeyValue instance. This is to set an upper boundary for a single entry saved in a storage file. Since they cannot be split it helps avoiding that a region cannot be split any further because the data is too large. It seems wise to set this to a fraction of the maximum region size. Setting it to zero or less disables the check.</p>
2970 </div>
2971 <div class="paragraph">
2972 <div class="title">Default</div>
2973 <p><code>10485760</code></p>
2974 </div>
2975 </dd>
2976 </dl>
2977 </div>
2978 <div id="hbase.server.keyvalue.maxsize" class="dlist">
2979 <dl>
2980 <dt class="hdlist1"><code>hbase.server.keyvalue.maxsize</code></dt>
2981 <dd>
2982 <div class="paragraph">
2983 <div class="title">Description</div>
2984 <p>Maximum allowed size of an individual cell, inclusive of value and all key components. A value of 0 or less disables the check. The default value is 10MB. This is a safety setting to protect the server from OOM situations.</p>
2985 </div>
2986 <div class="paragraph">
2987 <div class="title">Default</div>
2988 <p><code>10485760</code></p>
2989 </div>
2990 </dd>
2991 </dl>
2992 </div>
2993 <div id="hbase.client.scanner.timeout.period" class="dlist">
2994 <dl>
2995 <dt class="hdlist1"><code>hbase.client.scanner.timeout.period</code></dt>
2996 <dd>
2997 <div class="paragraph">
2998 <div class="title">Description</div>
2999 <p>Client scanner lease period in milliseconds.</p>
3000 </div>
3001 <div class="paragraph">
3002 <div class="title">Default</div>
3003 <p><code>60000</code></p>
3004 </div>
3005 </dd>
3006 </dl>
3007 </div>
3008 <div id="hbase.client.localityCheck.threadPoolSize" class="dlist">
3009 <dl>
3010 <dt class="hdlist1"><code>hbase.client.localityCheck.threadPoolSize</code></dt>
3011 <dd>
3012 <div class="paragraph">
3013 <div class="title">Default</div>
3014 <p><code>2</code></p>
3015 </div>
3016 </dd>
3017 </dl>
3018 </div>
3019 <div id="hbase.bulkload.retries.number" class="dlist">
3020 <dl>
3021 <dt class="hdlist1"><code>hbase.bulkload.retries.number</code></dt>
3022 <dd>
3023 <div class="paragraph">
3024 <div class="title">Description</div>
3025 <p>Maximum retries. This is maximum number of iterations to atomic bulk loads are attempted in the face of splitting operations 0 means never give up.</p>
3026 </div>
3027 <div class="paragraph">
3028 <div class="title">Default</div>
3029 <p><code>10</code></p>
3030 </div>
3031 </dd>
3032 </dl>
3033 </div>
3034 <div id="hbase.master.balancer.maxRitPercent" class="dlist">
3035 <dl>
3036 <dt class="hdlist1"><code>hbase.master.balancer.maxRitPercent</code></dt>
3037 <dd>
3038 <div class="paragraph">
3039 <div class="title">Description</div>
3040 <p>The max percent of regions in transition when balancing. The default value is 1.0. So there are no balancer throttling. If set this config to 0.01, It means that there are at most 1% regions in transition when balancing. Then the cluster&#8217;s availability is at least 99% when balancing.</p>
3041 </div>
3042 <div class="paragraph">
3043 <div class="title">Default</div>
3044 <p><code>1.0</code></p>
3045 </div>
3046 </dd>
3047 </dl>
3048 </div>
3049 <div id="hbase.balancer.period" class="dlist">
3050 <dl>
3051 <dt class="hdlist1"><code>hbase.balancer.period</code></dt>
3052 <dd>
3053 <div class="paragraph">
3054 <div class="title">Description</div>
3055 <p>Period at which the region balancer runs in the Master.</p>
3056 </div>
3057 <div class="paragraph">
3058 <div class="title">Default</div>
3059 <p><code>300000</code></p>
3060 </div>
3061 </dd>
3062 </dl>
3063 </div>
3064 <div id="hbase.normalizer.period" class="dlist">
3065 <dl>
3066 <dt class="hdlist1"><code>hbase.normalizer.period</code></dt>
3067 <dd>
3068 <div class="paragraph">
3069 <div class="title">Description</div>
3070 <p>Period at which the region normalizer runs in the Master.</p>
3071 </div>
3072 <div class="paragraph">
3073 <div class="title">Default</div>
3074 <p><code>300000</code></p>
3075 </div>
3076 </dd>
3077 </dl>
3078 </div>
3079 <div id="hbase.regions.slop" class="dlist">
3080 <dl>
3081 <dt class="hdlist1"><code>hbase.regions.slop</code></dt>
3082 <dd>
3083 <div class="paragraph">
3084 <div class="title">Description</div>
3085 <p>Rebalance if any regionserver has average + (average * slop) regions. The default value of this parameter is 0.001 in StochasticLoadBalancer (the default load balancer), while the default is 0.2 in other load balancers (i.e., SimpleLoadBalancer).</p>
3086 </div>
3087 <div class="paragraph">
3088 <div class="title">Default</div>
3089 <p><code>0.001</code></p>
3090 </div>
3091 </dd>
3092 </dl>
3093 </div>
3094 <div id="hbase.server.thread.wakefrequency" class="dlist">
3095 <dl>
3096 <dt class="hdlist1"><code>hbase.server.thread.wakefrequency</code></dt>
3097 <dd>
3098 <div class="paragraph">
3099 <div class="title">Description</div>
3100 <p>Time to sleep in between searches for work (in milliseconds). Used as sleep interval by service threads such as log roller.</p>
3101 </div>
3102 <div class="paragraph">
3103 <div class="title">Default</div>
3104 <p><code>10000</code></p>
3105 </div>
3106 </dd>
3107 </dl>
3108 </div>
3109 <div id="hbase.server.versionfile.writeattempts" class="dlist">
3110 <dl>
3111 <dt class="hdlist1"><code>hbase.server.versionfile.writeattempts</code></dt>
3112 <dd>
3113 <div class="paragraph">
3114 <div class="title">Description</div>
3115 <p>How many times to retry attempting to write a version file before just aborting. Each attempt is separated by the hbase.server.thread.wakefrequency milliseconds.</p>
3116 </div>
3117 <div class="paragraph">
3118 <div class="title">Default</div>
3119 <p><code>3</code></p>
3120 </div>
3121 </dd>
3122 </dl>
3123 </div>
3124 <div id="hbase.hregion.memstore.flush.size" class="dlist">
3125 <dl>
3126 <dt class="hdlist1"><code>hbase.hregion.memstore.flush.size</code></dt>
3127 <dd>
3128 <div class="paragraph">
3129 <div class="title">Description</div>
3130 <p>Memstore will be flushed to disk if size of the memstore exceeds this number of bytes. Value is checked by a thread that runs every hbase.server.thread.wakefrequency.</p>
3131 </div>
3132 <div class="paragraph">
3133 <div class="title">Default</div>
3134 <p><code>134217728</code></p>
3135 </div>
3136 </dd>
3137 </dl>
3138 </div>
3139 <div id="hbase.hregion.percolumnfamilyflush.size.lower.bound.min" class="dlist">
3140 <dl>
3141 <dt class="hdlist1"><code>hbase.hregion.percolumnfamilyflush.size.lower.bound.min</code></dt>
3142 <dd>
3143 <div class="paragraph">
3144 <div class="title">Description</div>
3145 <p>If FlushLargeStoresPolicy is used and there are multiple column families, then every time that we hit the total memstore limit, we find out all the column families whose memstores exceed a "lower bound" and only flush them while retaining the others in memory. The "lower bound" will be "hbase.hregion.memstore.flush.size / column_family_number" by default unless value of this property is larger than that. If none of the families have their memstore size more than lower bound, all the memstores will be flushed (just as usual).</p>
3146 </div>
3147 <div class="paragraph">
3148 <div class="title">Default</div>
3149 <p><code>16777216</code></p>
3150 </div>
3151 </dd>
3152 </dl>
3153 </div>
3154 <div id="hbase.hregion.preclose.flush.size" class="dlist">
3155 <dl>
3156 <dt class="hdlist1"><code>hbase.hregion.preclose.flush.size</code></dt>
3157 <dd>
3158 <div class="paragraph">
3159 <div class="title">Description</div>
3160 <p>If the memstores in a region are this size or larger when we go to close, run a "pre-flush" to clear out memstores before we put up the region closed flag and take the region offline. On close, a flush is run under the close flag to empty memory. During this time the region is offline and we are not taking on any writes. If the memstore content is large, this flush could take a long time to complete. The preflush is meant to clean out the bulk of the memstore before putting up the close flag and taking the region offline so the flush that runs under the close flag has little to do.</p>
3161 </div>
3162 <div class="paragraph">
3163 <div class="title">Default</div>
3164 <p><code>5242880</code></p>
3165 </div>
3166 </dd>
3167 </dl>
3168 </div>
3169 <div id="hbase.hregion.memstore.block.multiplier" class="dlist">
3170 <dl>
3171 <dt class="hdlist1"><code>hbase.hregion.memstore.block.multiplier</code></dt>
3172 <dd>
3173 <div class="paragraph">
3174 <div class="title">Description</div>
3175 <p>Block updates if memstore has hbase.hregion.memstore.block.multiplier times hbase.hregion.memstore.flush.size bytes. Useful preventing runaway memstore during spikes in update traffic. Without an upper-bound, memstore fills such that when it flushes the resultant flush files take a long time to compact or split, or worse, we OOME.</p>
3176 </div>
3177 <div class="paragraph">
3178 <div class="title">Default</div>
3179 <p><code>4</code></p>
3180 </div>
3181 </dd>
3182 </dl>
3183 </div>
3184 <div id="hbase.hregion.memstore.mslab.enabled" class="dlist">
3185 <dl>
3186 <dt class="hdlist1"><code>hbase.hregion.memstore.mslab.enabled</code></dt>
3187 <dd>
3188 <div class="paragraph">
3189 <div class="title">Description</div>
3190 <p>Enables the MemStore-Local Allocation Buffer, a feature which works to prevent heap fragmentation under heavy write loads. This can reduce the frequency of stop-the-world GC pauses on large heaps.</p>
3191 </div>
3192 <div class="paragraph">
3193 <div class="title">Default</div>
3194 <p><code>true</code></p>
3195 </div>
3196 </dd>
3197 </dl>
3198 </div>
3199 <div id="hbase.hregion.max.filesize" class="dlist">
3200 <dl>
3201 <dt class="hdlist1"><code>hbase.hregion.max.filesize</code></dt>
3202 <dd>
3203 <div class="paragraph">
3204 <div class="title">Description</div>