vmwarehadoop

Understand the default configuration

The stack provides a web panel to control the status of Hadoop. To access it, follow these steps:

Once inside authenticated, you will see a page like this:

WebUI

Hadoop configuration files

The Hadoop configuration files are located at /opt/bitnami/hadoop/etc/hadoop, and the most relevant ones are:

  • /opt/bitnami/hadoop/etc/hadoop/hadoop-env.sh and /opt/bitnami/hadoop/etc/hadoop/yarn-env.sh: Configuration options for the scripts found in /opt/bitnami/hadoop/bin.
  • /opt/bitnami/hadoop/etc/hadoop/APP-site.xml: Site-specific configuration for each Hadoop service.

Find more details about how to configure Hadoop settings in Hadoop’s official documentation.

Hadoop log files

The Hadoop log files for the specific services are created in the following directories:

  • /opt/bitnami/hadoop/logs: NameNode, Secondary NameNode, DataNode, Timeline server, History server, NodeManager and Resource Manager.
  • /opt/bitnami/hadoop/hive/logs: Derby, HiveServer2, Metastore and WebHCat.
  • /opt/bitnami/hadoop/hive/hcatalog/var/log: HCatalog.
  • /opt/bitnami/hadoop/pig/logs: Pig.

Hadoop ports

Each daemon in Hadoop listens to a different port. The most relevant ones are:

  • ResourceManager:
    • Scheduler: 8030.
    • Resource Tracker: 8031.
    • Service: 8032.
    • Web UI: 8088.
  • NodeManager:
    • Localizer: 8040.
    • Web UI: 8042.
  • Timeline Server:
    • Service: 10200.
    • Web UI: 8188.
  • History Server:
    • Service: 10020
    • Admin: 10033.
    • Web UI: 19888.
  • NameNode:
    • Service: 8020.
    • Web UI: 9870.
  • Secondary NameNode:
    • Web UI: 9868.
  • DataNode:
    • Data Transfer: 9866.
    • Service: 9867.
    • Web UI: 9864.
  • Hive:
    • Derby DB: 1527.
    • HCat/Metastore: 9083.
    • Hiveserver2 Thrift: 10000.
    • Hiveserver2 Web UI: 10002.
    • WebHCat: 50111.

IMPORTANT: Making the Hadoop YARN ports (particularly port 8088) public is a significant security risk. If, for development purposes, you need to access these ports from outside a trusted network, use a secure channel such as a VPN or an SSH tunnel.

All ports are closed by default. In order to access any of them, you have two options:

  • (Recommended) Create an SSH tunnel for accessing the port (refer to the FAQ for more information about SSH tunnels).
  • Open the port for remote access (refer to the FAQ for more information about opening ports).
Last modification February 14, 2019