Shell Cheatsheet - Commands

Getting help

--help

Most commands have built-in help info, usually with the --help or -h option, e.g. $ ls --help will give you more info about the command ls.

man

# Display manual of the command.
$ man $COMMAND

# Specify section number.
$ man 2 mount # the system call
$ man 8 mount # the admin command

# Display manual of man itself. Section info are included.
$ man man

# Search the man pages
$ man -k mount

# List all sections that have the term.
$ man -f mount
mount (8)            - mount a filesystem
mount (2)            - mount filesystem
...

• Section 1: user commands
• Section 2: system calls
• Section 3: library functions
• Section 4: special files
• Section 5: file formats
• Section 6: games
• Section 7: conventions and miscellany
• Section 8: administration and privileged commands
• Section L: math library functions
• Section N: tcl functions

Current Working Directory

ls lists CWD if given no other parameters.

Check CWD, these 2 are equivalent:

$ pwd # Print Working Directory
$ echo $PWD

More or Less

• more: can only go forward, bash autocompletion uses more
• less: can go backward, search, and more; almost everything else uses less

dd

The dd Unix utility program reads octet streams from a source to a destination, possibly performing data conversions in the process.

Create File With Zeroes

Creating a 1 MiB file, called foobar, filled with zeroes:

dd if=/dev/zero of=foobar count=1024 bs=1024

Note: The block size value can be given in SI (decimal) values, e.g. in GB, MB, etc. To create a 1 GB file one would simply type:

Test IO Performance

$ dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync
16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 3.94954 s, 272 MB/s

Read operations from /dev/zero return as many null characters (0x00) as requested in the read operation.

Unlike /dev/null, /dev/zero may be used as a source, not only as a sink for data.

dd if=/dev/zero of=/dev/<destination partition>

dd if=/dev/zero of=foobar count=1 bs=1GB

diff

diff
sdiff
vimdiff
colordiff

env

Filter:

$ env | grep HOME
$ env | egrep 'HOME|USER|VERSION|SHELL|PWD'

ldd

ldd is a powerful command-line tool that allows users to view an executable file's shared object dependencies.

ldd = List Dynamic Dependencies

$ ldd ./my-program
not a dynamic executable

$ ldd /usr/bin/gzip
linux-vdso.so.1 =>  (0x00007fff39fff000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fb842afa000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb842ea0000)

$ ldd /usr/bin/ssh
linux-vdso.so.1 =>  (0x00007fffd0164000)
libfipscheck.so.1 => /lib64/libfipscheck.so.1 (0x00007fb62de63000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fb62dc43000)
libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x00007fb62d8a9000)
...

LD_LIBRARY_PATH to include this path for the application to successfully run.

type

Shell Builtin

Linux built in commands:

$ type pwd cd
pwd is a shell builtin
cd is a shell builtin

And type itself is a shell builtin

$ type type
type is a shell builtin

Shell Keyword

Shell keywords are used in shell scripts

$ type if fi
if is a shell keyword
fi is a shell keyword

Alias

If you have alias defined in .bash_profile or .bashrc, for example

alias ls="ls -G"

where -G is set to enable colorized output. Then ls becomes an alias:

$ type ls
ls is aliased to `ls -G'

Executable

If you have docker installed, you will see something like this:

$ type docker
docker is /usr/local/bin/docker

Show all python executables

$ type -a python

xargs

Similar to map, apply functions on each item in the list

e.g. kill all ssh connections

$ ps -ax | grep ssh | cut -d ' ' -f 1 | xargs kill -9

tee

Reads the standard input and writes it to both the standard output and one or more files.

$ echo "test" | tee foo.txt
test
$ cat foo.txt
test

Locate a command

Use type or command to locate exact path for the env command:

$ type env
env is hashed (/usr/bin/env)

$ command -V env
env is hashed (/usr/bin/env)

which vs whereis

which: based on PATH

$ which hadoop
/Users/myname/lib/hadoop

whereis: based on standard binary directories

$ whereis hadoop
hadoop: /usr/bin/hadoop /etc/hadoop /usr/lib/hadoop /usr/share/man/man1/hadoop.1.gz

Check If Command Exists

$ command -v foo >/dev/null 2>&1
$ type foo >/dev/null 2>&1
$ hash foo 2>/dev/null

Example: add HADOOP_CLASSPATH if hadoop exists

command -v hadoop > /dev/null && {
    HADOOP_CLASSPATH=`hadoop classpath`
    CLASSPATH=$HADOOP_CLASSPATH:$CLASSPATH
}

TTY

TTY: teletype, now refers to any device that opens a physical or virtual terminal session.

Serial Port Terminals

Each serial port is considered to be a "device". e.g. /dev/ttys0

Pseudo Terminals

Pairs of devices such as /dev/ptyp3 and /dev/ttyp3; no physical device directly associated with either.

Controlling Terminal

/dev/tty

SSH to a Linux server (Ubuntu)

$ tty
/dev/pts/1

On a Mac

$ tty
/dev/ttys001

Shortcut

Change to different tty terminals: Ctrl + Alt + F1-F6

cd

cd to symlinked folder

cd -P /bin will go to /usr/bin

tree

Print directories tree

$ tree -dlL 2

• -d : show only the directories
• -l : follow symbolic links
• L 2 : show only level one and level two directories

Combine multiple commands

Change file extension (e.g. from .md to .mdx):

$ find . -type f -name "*.md" -exec rename 's/\.md$/.mdx/' '{}' \;

Text Processing

Useful tools:

• grep, egrep and fgrep: match patterns.
• sed(stream editor) is for programmatically editing files based on lines.
• awk is for text processing, especially useful for table-like text files like csv.

Replace characters

$ cat foo.txt | tr "," "_" > bar.txt

for unprintable character, e.g. \u0007, press ctrl-V ctrl-G

Delete character

Use tr -d, e.g. to remove all the o:

$ echo "Hello World" | tr -d o
Hell Wrld

Change Everything to Uppercase

$ cat foo.txt | tr "[a-z]" "[A-Z]"

Count Rows

$ cat foo.txt | wc -l

Extract a Column

$ echo 'a b c' | cut -d ' ' -f1
a

$ echo 'a b c' | cut -d ' ' -f2
b

Add - to list everything to the right

$ echo 'a b c' | cut -d ' ' -f2-
b c

split

Split data into chunks

Split by number of lines: split myfile, each chunk has 500 lines, prefixed by segment_, i.e. segment_aa, segment_ab, segment_ac...

$ split -l 500 myfile segment_

Split by size: split myfile, each chunk is 40k

$ split -b 40k myfile segment_

sort

• -k: (key) column number
• -t: delimiter

$ cat file | sort -nr -t \| -k 2 | head

FAQ

How to pretty print JSON?

$ cat data.json | python -m json.tool

How to add new line to the result?

Sometimes when decoding a base64 string, there's no newline at the end; the solution is to add && echo:

$ ... | base64 -d && echo

How to append Multiple Lines To a File

Use Here Document syntax, <<EOF means the multi-line text ends at string EOF

$ cat <<EOF
> foo
> bar
> EOF
foo
bar

e.g. add multiple lines to a file, like ~/.bashrc:

$ cat <<EOF >> ~/.bashrc
export KUBECONFIG=/path/to/kubeconfig
export FOO=foo
export BAR=bar
EOF

How to set and unset a env variable?

# set
$ export FOO="hello"

# unset
$ unset FOO

How to show history with timestamp?

history with timestamp: export HISTTIMEFORMAT="%F %T "

• %F: show the date in YYYY-MM-DD format.
• %T: show time in HH:MM:SS format.

How to check glibc version?

# Check ldd version.
$ ldd --version
ldd (Debian GLIBC 2.37-6+gl0) 2.37

# Check a built-in command, like ls.
$(ldd `which /bin/ls` | grep libc | awk '{print $3}')
GNU C Library (Debian GLIBC 2.37-6+gl0) stable release version 2.37.

chroot

chroot: changes the root file system directory as seen by a job, so that one program cannot access files outside of its directory tree. (for isolation)

chroot is both a system call and a wrapper program.

Cowsay

$ brew install cowsay
$ cowsay hello
 _______
< hello >
 -------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Also try:

$ fortune | cowsay
$ fortune | cowsay -f tux
$ cowsay -l

Shortcuts

grep | wc -> grep -c
sort | uniq | wc -> sort -u | wc

List the largest files/directories

If you want to find and print the top 10 largest files names (not directories) in a particular directory and its sub directories

$ find . -printf '%s %p\n'|sort -nr|head

To restrict the search to the present directory use -maxdepth 1 with find.

$ find . -maxdepth 1 -printf '%s %p\n'|sort -nr|head

print the top 10 largest "files and directories":

$ du -a . | sort -nr | head

e.g. check which log file/directory takes most usage

$ du -sh /var/log/* | sort -rh |less

Global Search And Replacement

$ find / -name game
$ find . -type f -name '*.txt' -exec sed -i '' s/this/that/g {} +

Count lines in markdown files

for file in `find ../src -name '*.md'`; do
  wc -l ${file}
done | sort -n > ../_tmp/result.txt

Change -l to -w to count words.

What is the double dash?

-- signifies the end of the options.

A double dash (--) is used in most Bash built-in commands and many other commands to signify the end of command options, after which only positional ("non-option") arguments are accepted.

For example: after the --, -v is no longer considered an option for kubectl

$ kubectl exec -ti $POD_NAME -- nginx -v
nginx version: nginx/1.25.3

What is a POSIX Shell?

The POSIX (the Portable Operating System Interface) shell is the standard Unix shell, i.e. it was formally defined and shipped in a published standard, it has many competing implementations on many different operating systems, but are compatible.

POSIX shell is basically Bourne shell, lives at the standardized location /bin/sh.

The POSIX standard does not recognize long flags like grep --file=FILE, but only the short flags like grep -f. (Because it does not define getopt_long function, only getopt function).

Other Commands

• sysctl - configure kernel parameters at runtime, not to be confused with systemctl.