Understand the default configuration
The stack provides a web panel to control the status of Hadoop. To access it, follow these steps:
- Go to http://localhost/cluster/.
- Authenticate yourself with the Administrator username and password.
Once inside authenticated, you will see a page like this:
Hadoop configuration files
The Hadoop configuration files are located at installdir/hadoop/etc/hadoop, and the most relevant ones are:
- installdir/hadoop/etc/hadoop/hadoop-env.sh and installdir/hadoop/etc/hadoop/yarn-env.sh: Configuration options for the scripts found in installdir/hadoop/bin.
- installdir/hadoop/etc/hadoop/APP-site.xml: Site-specific configuration for each Hadoop service.
Find more details about how to configure Hadoop settings in Hadoop’s official documentation.
Hadoop log files
The Hadoop log files for the specific services are created in the following directories:
- installdir/hadoop/logs: NameNode, Secondary NameNode, DataNode, Timeline server, History server, NodeManager and Resource Manager.
- installdir/hadoop/hive/logs: Derby, HiveServer2, Metastore and WebHCat.
- installdir/hadoop/hive/hcatalog/var/log: HCatalog.
- installdir/hadoop/pig/logs: Pig.
Hadoop ports
Each daemon in Hadoop listens to a different port. The most relevant ones are:
- ResourceManager:
- Scheduler: 8030.
- Resource Tracker: 8031.
- Service: 8032.
- Web UI: 8088.
- NodeManager:
- Localizer: 8040.
- Web UI: 8042.
- Timeline Server:
- Service: 10200.
- Web UI: 8188.
- History Server:
- Service: 10020
- Admin: 10033.
- Web UI: 19888.
- NameNode:
- Service: 8020.
- Web UI: 9870.
- Secondary NameNode:
- Web UI: 9868.
- DataNode:
- Data Transfer: 9866.
- Service: 9867.
- Web UI: 9864.
- Hive:
- Derby DB: 1527.
- HCat/Metastore: 9083.
- Hiveserver2 Thrift: 10000.
- Hiveserver2 Web UI: 10002.
- WebHCat: 50111.
IMPORTANT: Making the Hadoop YARN ports (particularly port 8088) public is a significant security risk. If, for development purposes, you need to access these ports from outside a trusted network, use a secure channel such as a VPN or an SSH tunnel.