nativeInstallerhadoop

Understand the default configuration

Before running the commands shown on this page, you should load the Bitnami stack environment by executing the installdir/use_APPNAME script (Linux and MacOS) or by clicking the shortcut in the Start Menu under “Start -> Bitnami APPNAME Stack -> Application console” (Windows). On OS X VMs, the installation directory is /opt/bitnami and OS X VM users can click the “Open Terminal” button to run commands. Learn more about the Bitnami stack environment and about OS X VMs.

The stack provides a web panel to control the status of Hadoop. To access it, follow these steps:

Once inside authenticated, you will see a page like this:

WebUI

Hadoop configuration files

The Hadoop configuration files are located at installdir/hadoop/etc/hadoop, and the most relevant ones are:

  • installdir/hadoop/etc/hadoop/hadoop-env.sh and installdir/hadoop/etc/hadoop/yarn-env.sh: Configuration options for the scripts found in installdir/hadoop/bin.
  • installdir/hadoop/etc/hadoop/APP-site.xml: Site-specific configuration for each Hadoop service.

Find more details about how to configure Hadoop settings in Hadoop’s official documentation.

Hadoop log files

The Hadoop log files for the specific services are created in the following directories:

  • installdir/hadoop/logs: NameNode, Secondary NameNode, DataNode, Timeline server, History server, NodeManager and Resource Manager.
  • installdir/hadoop/hive/logs: Derby, HiveServer2, Metastore and WebHCat.
  • installdir/hadoop/hive/hcatalog/var/log: HCatalog.
  • installdir/hadoop/pig/logs: Pig.

Hadoop ports

Each daemon in Hadoop listens to a different port. The most relevant ones are:

  • ResourceManager:
    • Scheduler: 8030.
    • Resource Tracker: 8031.
    • Service: 8032.
    • Web UI: 8088.
  • NodeManager:
    • Localizer: 8040.
    • Web UI: 8042.
  • Timeline Server:
    • Service: 10200.
    • Web UI: 8188.
  • History Server:
    • Service: 10020
    • Admin: 10033.
    • Web UI: 19888.
  • NameNode:
    • Service: 8020.
    • Web UI: 9870.
  • Secondary NameNode:
    • Web UI: 9868.
  • DataNode:
    • Data Transfer: 9866.
    • Service: 9867.
    • Web UI: 9864.
  • Hive:
    • Derby DB: 1527.
    • HCat/Metastore: 9083.
    • Hiveserver2 Thrift: 10000.
    • Hiveserver2 Web UI: 10002.
    • WebHCat: 50111.

IMPORTANT: Making the Hadoop YARN ports (particularly port 8088) public is a significant security risk. If, for development purposes, you need to access these ports from outside a trusted network, use a secure channel such as a VPN or an SSH tunnel.

Last modification April 8, 2020