Common Big Data Commands

Linux (vi/vim)

Normal Mode

Syntax	Description
yy	Copy the current line
yy	Copy a block (from line X to line Y)
p	Move cursor to target line and paste
u	Undo the last action
dd	Delete the current line
dd	Delete the current line and the following lines
x	Delete one character, equivalent to del
X	Delete one character, equivalent to Backspace
yw	Copy a word
dw	Delete a word
shift+^	Move to the beginning of the line
shift+$	Move to the end of the line
1+shift+g	Move to the top of the file, number
shift+g	Move to the bottom of the file
N+shift+g	Move to the target line

Edit Mode

Key	Description
i	Insert before the cursor
a	Insert after the cursor
o	Insert a new line below the current line
I	Insert at the beginning of the current line
A	Insert at the end of the current line
O	Insert a new line above the current line

Command Mode

Command	Description
:w	Save
:q	Quit
:!	Force execute
/	n searches next, N searches previous
?	n searches previous, shift+n searches next
:set nu	Show line numbers
:set nonu	Hide line numbers

Compression and Decompression

gzip/gunzip Compression

(1) Can only compress files, not directories

(2) Does not keep the original file

gzip compression: gzip hello.txt

gunzip decompression: gunzip hello.txt.gz

zip/unzip Compression

Can compress directories and keeps the source files

zip compression (compresses 1.txt and 2.txt, the compressed file is named mypackage.zip): zip hello.zip hello.txt world.txt

unzip decompression: unzip hello.zip

unzip decompress to a specific directory: unzip hello.zip -d /opt

tar Archiving

tar compress multiple files: tar -zcvf hello.txt world.txt

tar compress a directory: tar -zcvf hello.tar.gz opt/

tar decompress to the current directory: tar -zxvf hello.tar.gz

tar decompress to a specific directory: tar -zxvf hello.tar.gz -C /opt

RPM

RPM query command: rpm -qa |grep firefox

RPM uninstall command:

rpm -e xxxxxx

rpm -e --nodeps xxxxxx` (does not check dependencies)

RPM install command:

rpm -ivh xxxxxx.rpm

rpm -ivh --nodeps fxxxxxx.rpm` (--nodeps, does not check dependency progress)

Option	Description
-i	-i=install, install
-v	-v=verbose, show detailed information
-h	-h=hash, progress bar
--nodeps	--nodeps, does not check dependency progress

---

Shell

Input/Output Redirection

Command	Description
command > file	Redirect output to file
command < file	Redirect input to file
command >> file	Redirect output to file in append mode
n > file	Redirect the file with file descriptor n to file
n >> file	Redirect the file with file descriptor n to file in append mode
n >& m	Merge output files m and n
n <& m	Merge input files m and n
<< tag	Use the content between the start tag and end tag as input

Script Editing

Shortcut	Description
shift	Shift parameters left
$@	All parameters
$#	Number of parameters

---

Hadoop

Startup Commands

Description	Command Script
Start HDFS cluster	sbin/start-dfs.sh
Start YARN	sbin/start-yarn.sh

hadoop fs/hdfs dfs Commands

Description	Command
Create directory	hdfs dfs -mkdir -p /data/flink
List directory	hdfs dfs -ls /
Copy from HDFS to local	hdfs dfs -copyToLocal /data/data.txt ./
Upload file to cluster (from local)	hdfs dfs -copyFromLocal data.txt /
Download file	hdfs dfs -get /data/flink
Delete file from cluster	hdfs dfs -rm /data/flink
Delete folder	hdfs dfs -rm -r -skipTrash /data
Move (cut and paste) from local to HDFS	hdfs dfs -moveFromLocal data.txt /data/
Append a file to the end of an existing file	hdfs dfs -appendToFile data1.txt /data/data.txt
Display file content	hdfs dfs -cat data.txt
Change file permissions	hdfs dfs -chmod 777 xxx.sh
Change file owner and group	hdfs dfs -chown root:root data.txt
Copy from one HDFS path to another HDFS path	hdfs dfs -cp data.txt /data1.txt
Move file within HDFS directory	hdfs dfs -mv data.txt /opt/
Merge and download multiple files	hdfs dfs -getmerge /data/* ./data_merge.txt
hadoop fs -put	Equivalent to copyFromLocal
Display the end of a file	hdfs dfs -tail data.txt
Delete file or folder	hdfs dfs -rm /data/data.txt
Delete empty directory	hdfs dfs -rmdir /data
Show size information of a folder	hdfs dfs -du -s -h /data
Show file size information under a folder	hdfs dfs -du -h /data
Set the number of replicas for a file in HDFS	hdfs dfs -setrep 3 /data/data.txt

yarn Commands

Description	Command
List running YARN tasks	yarn application -list appID
Kill a YARN task by ID	yarn application -kill appID
View task log information	yarn logs -applicationId appID

---

Zookeeper

Startup Commands

Description	Command Script
Start Zookeeper service	zkServer.sh start
Check Zookeeper status	zkServer.sh status
Stop Zookeeper service	zkServer.sh stop
Start Zookeeper client	zkCli.sh -server 127.0.0.1:2181
Exit Zookeeper client	quit

Basic Operations

Description	Command Script
List contents of the current znode	ls /
Create a normal node (path first, value second)	create /bigdata/flink "flink"
Get the value of a node	get /bigdata
Set the value of a node	set /bigdata/flink "flinksql"
Delete a node	delete /bigdata/flink
Recursively delete a node	rmr /bigdata

Four-Letter Commands

Command	Description	Example
conf	Detailed configuration information of the ZK service	echo conf | nc 127.0.0.1 2181
stat	Brief information about the client connection to ZK	See above
srvr	Detailed information about the ZK service	See above
cons	Detailed information about client connections to ZK	See above
mntr	Current performance status of the ZK service	See above
crst	Reset all current connections and sessions	See above
dump	List unprocessed sessions and connection information	See above
envi	List ZK version info, hostname, Java version, server name, etc.	See above
ruok	Test if the server is running; returns imok if running, otherwise empty	See above
srst	Reset all statistics of Zookeeper	See above
wchs	List total number of watches, connections	See above
wchp	List all watch paths and session IDs	See above
mntr	List key performance data of the cluster, including ZK version, node count, ephemeral node count, etc.	See above

---

Kafka

Note: I only list one machine here. You can also use the commands with ./bin/xx.sh (e.g., ./bin/kafka-topics.sh).

List all topics in the current server

kafka-topics --zookeeper xxxxxx:2181 --list --exclude-internal 

Explanation:

exclude-internal: exclude Kafka internal topics

Example: --exclude-internal --topic "test_.*"

Create a topic

kafka-topics --zookeeper xxxxxx:2181 --create 
--replication-factor 
--partitions 1 
--topic topic_name

Explanation:

--topic defines the topic name

--replication-factor defines the number of replicas

--partitions defines the number of partitions

Delete a topic

Note: You need to set delete.topic.enable=true in server.properties, otherwise it will only be marked for deletion.

kafka-topics --zookeeper xxxxxx:2181 --delete --topic topic_name

Producer

kafka-console-producer --broker-list xxxxxx:9092 --topic topic_name

Optional: --property parse.key=true (for messages with keys)

Consumer

kafka-console-consumer --bootstrap-server xxxxxx:9092 --topic topic_name

Note: Optional parameters

--from-beginning: reads all historical data from the topic

--whitelist '.*': consumes all topics

--property print.key=true: displays the key during consumption

--partition 0: consume from a specific partition

--offset: consume from a specific starting offset

View details of a specific topic

kafka-topics --zookeeper xxxxxx:2181 --describe --topic topic_name

Modify the number of partitions

kafka-topics --zookeeper xxxxxx:2181 --alter --topic topic_name --partitions 6

View information about a specific consumer group

kafka-consumer-groups --bootstrap-server xxxxxx:9092 --describe --group group_name 

Delete a consumer group

kafka-consumer-groups --bootstrap-server xxxxxx:9092 --delete --group group_name 

Reset offset

kafka-consumer-groups --bootstrap-server xxxxxx:9092 --group group_name

--reset-offsets --all-topics --to-latest --execute 

Leader re-election

Re-elect the Leader for a specific topic and partition using PREFERRED: preferred replica strategy

kafka-leader-election --bootstrap-server xxxxxx:9092 
--topic topic_name --election-type PREFERRED --partition 0

Re-elect the Leader for all topics and partitions using PREFERRED: preferred replica strategy

kafka-leader-election --bootstrap-server xxxxxx:9092 
--election-type preferred --all-topic-partitions

Query Kafka version information

kafka-configs --bootstrap-server xxxxxx:9092
--describe --version

Add, delete, and modify configurations

Description	Parameter
Select type	--entity-type (topics/clients/users/brokers/broker-loggers)
Type name	--entity-name
Delete config	--delete-config k1=v1,k2=v2
Add/Modify config	--add-config k1,k2

Add/modify dynamic configuration for a topic

kafka-configs --bootstrap-server xxxxxx:9092
--alter --entity-type topics --entity-name topic_name 
--add-config file.delete.delay.ms=222222,retention.ms=999999

Delete dynamic configuration for a topic

kafka-configs --bootstrap-server xxxxxx:9092 
--alter --entity-type topics --entity-name topic_name 
--delete-config file.delete.delay.ms,retention.ms

Continuously batch pull messages

Consume a maximum of 10 messages at a time (without parameters, it means continuous consumption)

kafka-verifiable-consumer --bootstrap-server xxxxxx:9092 
--group group_name
--topic topic_name --max-messages 10

Delete messages from a specific partition

Delete messages from a specific partition of a specific topic up to offset 1024

json file offset-json-file.json

{
    "partitions": [
        {
            "topic": "topic_name",
            "partition": 0,
            "offset": 1024
        }
    ],
    "version": 1
}
kafka-delete-records --bootstrap-server xxxxxx:9092 
--offset-json-file offset-json-file.json

View Broker disk information

Query disk information for a specific topic

kafka-log-dirs --bootstrap-server xxxxxx:9090 
--describe --topic-list topic1,topic2

Query disk information for a specific Broker

kafka-log-dirs --bootstrap-server xxxxxx:9090 
--describe --topic-list topic1 --broker-list 0

Hive

Startup Commands

Description	Command
Start hiveserver2 service	bin/hiveserver2
Start beeline	bin/beeline
Connect to hiveserver2	beeline> !connect jdbc:hive2://hadoop102:10000
Metastore service	bin/hive --service metastore

Hive script to start metadata services (metastore and hiveserver2) and gracefully shut down

Start: hive.sh start
Stop: hive.sh stop
Restart: hive.sh restart
Status: hive.sh status

The script is as follows:

#!/bin/bash
HIVE_LOG_DIR=$HIVE_HOME/logs

mkdir -p $HIVE_LOG_DIR

#Check if the process is running normally, parameter 1 is the process name, parameter 2 is the process port
function check_process()
{
    pid=$(ps -ef 2>/dev/null | grep -v grep | grep -i $1 | awk '{print $2}')
    ppid=$(netstat -nltp 2>/dev/null | grep $2 | awk '{print $7}' | cut -d '/' -f 1)
    echo $pid
    [[ "$pid" =~ "$ppid" ]] && [ "$ppid" ] && return 0 || return 1
}

function hive_start()
{
    metapid=$(check_process HiveMetastore 9083)
    cmd="nohup hive --service metastore >$HIVE_LOG_DIR/metastore.log 2>&1 &"
    cmd=$cmd" sleep4; hdfs dfsadmin -safemode wait >/dev/null 2>&1"
    [ -z "$metapid" ] && eval $cmd || echo "Metastroe service already started"
    server2pid=$(check_process HiveServer2 10000)
    cmd="nohup hive --service hiveserver2 >$HIVE_LOG_DIR/hiveServer2.log 2>&1 &"
    [ -z "$server2pid" ] && eval $cmd || echo "HiveServer2 service already started"
}

function hive_stop()
{
    metapid=$(check_process HiveMetastore 9083)
    [ "$metapid" ] && kill $metapid || echo "Metastore service not started"
    server2pid=$(check_process HiveServer2 10000)
    [ "$server2pid" ] && kill $server2pid || echo "HiveServer2 service not started"
}

case $1 in
"start")
    hive_start
    ;;
"stop")
    hive_stop
    ;;
"restart")
    hive_stop
    sleep 2
    hive_start
    ;;
"status")
    check_process HiveMetastore 9083 >/dev/null && echo "Metastore service is running normally" || echo "Metastore service is running abnormally"
    check_process HiveServer2 10000 >/dev/null && echo "HiveServer2 service is running normally" || echo "HiveServer2 service is running abnormally"
    ;;
*)
    echo Invalid Args!
    echo 'Usage: '$(basename $0)' start|stop|restart|status'
    ;;
esac

Common Interactive Commands

Description	Command
Execute SQL without entering the Hive interactive window	bin/hive -e "sql statement"
Execute SQL statements from a script	bin/hive -f hive.sql
Exit the Hive window	exit or quit
View HDFS file system from the command window	dfs -ls /
View local file system from the command window	! ls /data/h

SQL (Special)

Description	Statement
List all databases in Hive	show databases
Use the default database	use default
View table structure	desc table_name
View databases	show databases
Rename a table	alter table table1 rename to table2
Modify a field in a table	alter table table_name change name user_name String
Modify a field type	alter table table_name change salary salary Double
Create an external table	create external table ....
View external table information	desc formatted outsidetable
Create a view	create view view_name as select * from table_name .....
Add data	load data local inpath 'xxx' overwrite into table table_name partition(day='2021-12-01')

Built-in Functions

(1) NVL

Assigns a value to NULL data. Its format is NVL(value, default_value). If value is NULL, the NVL function returns default_value; otherwise, it returns value. If both parameters are NULL, it returns NULL.

select nvl(column, 0) from xxx；

(2) Row to Column

Function	Description
CONCAT(string A/col, string B/col…)	Returns the concatenated result of the input strings, supports any number of input strings
CONCAT_WS(separator, str1, str2,...)	The first parameter is the separator between parameters. If the separator is NULL, the return value is also NULL. This function skips any NULL and empty strings after the separator parameter. The separator is added between the concatenated strings.
COLLECT_SET(col)	Deduplicates and aggregates the values of a field, producing an array type field
COLLECT_LIST(col)	This function only accepts basic data types. Its main function is to aggregate the values of a field without deduplication, producing an array type field.

(3) Column to Row (One column to multiple rows)

Split(str, separator): Splits the string according to the following separator, converting it into a character array.

EXPLODE(col): Splits a complex array or map structure in a Hive column into multiple rows.

LATERAL VIEW

Usage:

LATERAL VIEW udtf(expression) tableAlias AS columnAlias

Explanation: lateral view is used together with UDTFs like split, explode, etc. It can split one row of data into multiple rows, and on this basis, aggregation can be performed on the split data.

lateral view first calls the UDTF for each row of the original table. The UDTF splits one row into one or more rows. lateral view then combines the results to produce a virtual table that supports table aliases.

Prepare test data source

movie	category
《功勋》	记录,剧情
《战狼2》	战争,动作,灾难

SQL

SELECT movie,category_name 
FROM movie_info 
lateral VIEW
explode(split(category,",")) movie_info_tmp  AS category_name ;

Test Results

《功勋》      记录
《功勋》      剧情
《战狼2》     战争
《战狼2》     动作
《战狼2》     灾难

Window Functions

(1) OVER()

Defines the data window size for the analysis function to work on. This data window size may change as rows change.

(2) CURRENT ROW

n PRECEDING: n rows of data before
n FOLLOWING: n rows of data after

(3) UNBOUNDED

UNBOUNDED PRECEDING: no boundary before, meaning from the starting point
UNBOUNDED FOLLOWING: no boundary after, meaning to the ending point

SQL Example: Aggregation from the starting point to the current row

select 
    sum(money) over(partition by user_id order by pay_time rows between UNBOUNDED PRECEDING and current row) 
from or_order;

SQL Example: Aggregation of the current row and the previous row

select 
    sum(money) over(partition by user_id order by pay_time rows between 1 PRECEDING and current row) 
from or_order;

SQL Example: Aggregation of the current row, the previous row, and the next row

select 
    sum(money) over(partition by user_id order by pay_time rows between 1 PRECEDING AND 1 FOLLOWING )
from or_order;

SQL Example: The current row and all subsequent rows

select 
    sum(money) over(partition by user_id order by pay_time rows between current row and UNBOUNDED FOLLOWING  )
from or_order;

(4) LAG(col,n,default_val)

The nth row before the current row; if it doesn't exist, use default_val.

(5) LEAD(col,n, default_val)

The nth row after the current row; if it doesn't exist, use default_val.

SQL Example: Query user purchase details along with the previous and next purchase times

select 
 user_id,,pay_time,money,
 
 lag(pay_time,1,'1970-01-01') over(PARTITION by name order by pay_time) prev_time,
 
 lead(pay_time,1,'1970-01-01') over(PARTITION by name order by pay_time) next_time
from or_order;

(6) FIRST_VALUE(col,true/false)

The first value in the current window. If the second parameter is true, skip null values.

(7) LAST_VALUE (col,true/false)

The last value in the current window. If the second parameter is true, skip null values.

SQL Example: Query the user's first purchase time and last purchase time for each month

select
 FIRST_VALUE(pay_time) 
     over(
         partition by user_id,month(pay_time) order by pay_time 
         rows between UNBOUNDED PRECEDING and UNBOUNDED FOLLOWING
         ) first_time,
 
 LAST_VALUE(pay_time) 
     over(partition by user_id,month(pay_time) order by pay_time rows between UNBOUNDED PRECEDING and UNBOUNDED FOLLOWING
     ) last_time
from or_order;

(8) NTILE(n)

Distributes the rows of an ordered window into a specified number of groups. Each group has a number starting from 1. For each row, NTILE returns the number of the group to which this row belongs. (Used to split grouped data into n slices in order, returning the current slice value.)

SQL Example: Query order information for the first 25% of time

select * from (
    select User_id,pay_time,money,
    
    ntile(4) over(order by pay_time) sorted
    
    from or_order
) t
where sorted = 1;

The 4 By's

(1) Order By

Global sorting, only one Reducer.

(2) Sort By

Ordered within partitions.

(3) Distrbute By

Similar to Partition in MR, used for partitioning, combined with sort by.

(4) Cluster By

When the Distribute by and Sort by fields are the same, you can use the Cluster by method. Cluster by has the functionality of both Distribute by and Sort by. However, sorting can only be ascending; you cannot specify the sort order as ASC or DESC.

In production environments, Order By is used less frequently as it can easily lead to OOM.

In production environments, Sort By + Distrbute By is used more often.

Sorting Functions

(1) RANK()

Duplicates when sorting is the same, the total count does not change.

1
1
3
3
5

(2) DENSE_RANK()

Duplicates when sorting is the same, the total count decreases.

1
1
2
2
3

(3) ROW_NUMBER()

Calculates based on order.

1
2
3
4
5

Date Functions

datediff: Returns the number of days from the end date minus the start date.

datediff(string enddate, string startdate) 

select datediff('2021-11-20','2021-11-22') 

date_add: Returns the date after adding days days to the start date.

date_add(string startdate, int days) 

select date_add('2021-11-20',3) 

date_sub: Returns the date after subtracting days days from the start date.

date_sub (string startdate, int days) 

select date_sub('2021-11-22',3)

---

Redis

Startup Commands

key

Command	Description
keys *	View all keys in the current database
exists	Check if a key exists
type	View the type of a key
del	Delete a key
expire	Set expiration time for a key-value pair, in seconds
ttl	View how long until expiration; -1 means never expires, -2 means already expired
dbsize	View the number of keys in the current database
flushdb	Clear the current database
Flushall	Clear all databases

String

Command	Description
get	Query the corresponding key value
set	Add a key-value pair
append	Append the given value to the end of the original value
strlen	Get the length of the value
setnx	Set the value of the key only if the key does not exist
incr	Increment the numeric value stored in the key by 1. Can only operate on numeric values. If empty, the new value is 1.
decr	Decrement the numeric value stored in the key by 1. Can only operate on numeric values. If empty, the new value is -1.
incrby / decrby	Increment or decrement the numeric value stored in the key by a custom step
mset	Set one or more key-value pairs simultaneously
mget	Get one or more values simultaneously
msetnx	Set one or more key-value pairs simultaneously, only if all given keys do not exist
getrange	Get a range of the value, similar to substring in Java
setrange	Overwrite the stored string value starting from
setex	Set the key-value pair and expiration time simultaneously, in seconds
getset	Replace old with new; set a new value while getting the old value

List

Command	Description
lpush/rpush	Insert one or more values from the left/right.
lpop/rpop	Pop a value from the left/right. Value exists when key exists; key dies when value is gone.
rpoplpush	Pop a value from the right side of a list and insert it into the left side of another list
lrange	Get elements by index (from left to right)
lindex	Get an element by index (from left to right)
llen	Get the length of the list
linsert before/after	Insert a value before/after a specified element
lrem	Delete n values from the left (from left to right)

Set

Command	Description
sadd....	Add one or more member elements to the set key. Members that already exist in the set are ignored.
smembers	Get all values of the set.
sismember	Check if the set contains the value; returns 1 if yes, 0 if no
scard	Return the number of elements in the set.
srem....	Delete a specific element from the set.
spop	Randomly pop a value from the set.
srandmember	Randomly get n values from the set. Does not remove them from the set.
sinter	Return the intersection elements of two sets.
sunion	Return the union elements of two sets.
sdiff	Return the difference elements of two sets.

Hash

Command	Description
hset	Assign a value to a key in the hash
hget	Get a value from the hash
hmset...	Batch set hash values
hexists key	Check if a given field exists in the hash table key.
hkeys	List all fields of the hash set
hvals	List all values of the hash set
hincrby	Add an increment to the value of a field in the hash table key
hsetnx	Set the value of a field in the hash table key only if the field does not exist

zset (Sorted set)

Command	Description
zadd...	Add one or more member elements and their score values to the sorted set key
zrange [WITHSCORES]	Return the elements in the sorted set key within the specified index range. With WITHSCORES, scores are returned along with the values.
zrangebyscore key min max [withscores] [limit offset count]	Return all members in the sorted set key with a score between min and max (including min and max). Members are sorted by score in ascending order.
zrevrangebyscore key max min [withscores] [limit offset count]	Same as above, but sorted in descending order.
zincrby	Add an increment to the score of an element
zrem	Delete the element with the specified value from the set
zcount	Count the number of elements in the set within a score range
zrank	Return the rank of the value in the set, starting from 0.

---

Flink

Startup

./start-cluster.sh 

run

./bin/flink run [OPTIONS]

./bin/flink run -m yarn-cluster -c com.wang.flink.WordCount /opt/app/WordCount.jar

OPTIONS	Description
-d	detached: whether to use detached mode
-m	jobmanager: specify the jobmanager to submit to
-yat	--yarnapplicationType: set the type of the YARN application
-yD	Use the value of the given property
-yd	--yarndetached: use YARN detached mode
-yh	--yarnhelp: help for YARN session
-yid	--yarnapplicationId: attach to a running YARN session
-yj	--yarnjar: path to the Flink jar file
-yjm	--yarnjobManagerMemory: memory for the jobmanager (in MB)
-ynl	--yarnnodeLabel: specify the YARN node label for the YARN application
-ynm	--yarnname: customize the YARN application name
-yq	--yarnquery: display available YARN resources
-yqu	--yarnqueue: specify the YARN queue
-ys	--yarnslots: specify the number of slots per taskmanager
-yt	yarnship: transfer files in the specified directory
-ytm	--yarntaskManagerMemory: memory for each taskmanager
-yz	--yarnzookeeperNamespace: namespace for creating ZK subpaths for HA
-z	--zookeeperNamespace: namespace for creating ZK subpaths for HA
-p	Parallelism
-yn	Number of YARN containers to allocate (= number of task managers)

info

./bin/flink info [OPTIONS]

OPTIONS	Description
-c	Program entry point, main class
-p	Parallelism

list

./bin/flink list [OPTIONS]

OPTIONS	Description
-a	--all: show all applications and corresponding job IDs
-r	--running: show running applications and job IDs
-s	--scheduled: show scheduled applications and job IDs
-m	--jobmanager: specify the jobmanager to connect to
-yid	--yarnapplicationId: attach to the YARN session corresponding to the specified YARN ID
-z	--zookeeperNamespace: namespace for creating ZK subpaths for HA

stop

./bin/flink stop  [OPTIONS] <Job ID>

OPTIONS	Description
-d	Send MAX_WATERMARK before taking a savepoint and stopping the pipeline
-p	savepointPath: path for the savepoint 'xxxxx'
-m	--jobmanager: specify the jobmanager to connect to
-yid	--yarnapplicationId: attach to the YARN session corresponding to the specified YARN ID
-z	--zookeeperNamespace: namespace for creating ZK subpaths for HA

cancel (deprecated)

./bin/flink cancel  [OPTIONS] <Job ID>

OPTIONS	Description
-s	Use "stop" instead
-D	Allows specifying multiple general configuration options
-m	Address of the JobManager to connect to
-yid	--yarnapplicationId: attach to the YARN session corresponding to the specified YARN ID
-z	--zookeeperNamespace: namespace for creating ZK subpaths for