UNIX

Commands

Files

  • ls --- lists your files
    ls -l --- lists your files in 'long format', which contains lots of useful information, e.g. the exact size of the file, who owns the file and who has the right to look at it, and when it was last modified.
    ls -a --- lists all files, including the ones whose filenames begin in a dot, which you do not always want to see.
    There are many more options, for example to list files by size, by date, recursively etc.
  • more filename --- shows the first part of a file, just as much as will fit on one screen. Just hit the space bar to see more or q to quit. You can use /pattern to search for a pattern.
  • emacs filename --- is an editor that lets you create and edit a file. See the [emacs page][3].
  • mv filename1 filename2 --- moves a file (i.e. gives it a different name, or moves it into a different directory (see below)
  • cp filename1 filename2 --- copies a file
  • rm filename --- removes a file. It is wise to use the option rm -i, which will ask you for confirmation before actually deleting anything. You can make this your default by making an [alias][4] in your .cshrc file.
  • diff filename1 filename2 --- compares files, and shows where they differ
  • wc filename --- tells you how many lines, words, and characters there are in a file
  • chmod options filename --- lets you change the read, write, and execute permissions on your files. The default is that only you can look at them and change them, but you may sometimes want to change these permissions. For example, chmod o+r filename will make the file readable for everyone, and chmod o-r filename will make it unreadable for others again. Note that for someone to be able to actually look at the file the directories it is in need to be at least executable. See [help protection][5] for more details.
  • File Compression
    • gzip filename --- compresses files, so that they take up much less space. Usually text files compress to about half their original size, but it depends very much on the size of the file and the nature of the contents. There are other tools for this purpose, too (e.g. compress), but gzip usually gives the highest compression rate. Gzip produces files with the ending '.gz' appended to the original filename.
    • gunzip filename --- uncompresses files compressed by gzip.
    • gzcat filename --- lets you look at a gzipped file without actually having to gunzip it (same as gunzip -c). You can even print it directly, using gzcat filename | lpr
  • printing
    • lpr filename --- print. Use the -P option to specify the printer name if you want to use a printer other than your default printer. For example, if you want to print double-sided, use 'lpr -Pvalkyr-d', or if you're at CSLI, you may want to use 'lpr -Pcord115-d'. See 'help printers' for more information about printers and their locations.
    • lpq --- check out the printer queue, e.g. to get the number needed for removal, or to see how many other files will be printed before yours will come out
    • lprm jobnumber --- remove something from the printer queue. You can find the job number by using lpq. Theoretically you also have to specify a printer name, but this isn't necessary as long as you use your default printer in the department.
    • genscript --- converts plain text files into postscript for printing, and gives you some options for formatting. Consider making an alias like alias ecop 'genscript -2 -r !* | lpr -h -Pvalkyr' to print two pages on one piece of paper.
    • dvips filename --- print .dvi files (i.e. files produced by LaTeX). You can use dviselect to print only selected pages. See the [LaTeX page][6] for more information about how to save paper when printing drafts.

Directories

Directories, like folders on a Macintosh, are used to group files together in a hierarchical structure.

  • mkdir dirname --- make a new directory
  • cd dirname --- change directory. You basically 'go' to another directory, and you will see the files in that directory when you do 'ls'. You always start out in your 'home directory', and you can get back there by typing 'cd' without arguments. 'cd ..' will get you one level up from your current position. You don't have to walk along step by step - you can make big leaps or avoid walking around by specifying [pathnames][7].
  • pwd --- tells you where you currently are.

Finding things

  • ff --- find files anywhere on the system. This can be extremely useful if you've forgotten in which directory you put a file, but do remember the name. In fact, if you use ff -p you don't even need the full name, just the beginning. This can also be useful for finding other things on the system, e.g. documentation.
  • grep string filename(s) --- looks for the string in the files. This can be useful a lot of purposes, e.g. finding the right file among many, figuring out which is the right version of something, and even doing serious corpus work. grep comes in several varieties (grep, egrep, and fgrep) and has a lot of very flexible options. Check out the man pages if this sounds good to you.

About other people

  • w --- tells you who's logged in, and what they're doing. Especially useful: the 'idle' part. This allows you to see whether they're actually sitting there typing away at their keyboards right at the moment.
  • who --- tells you who's logged on, and where they're coming from. Useful if you're looking for someone who's actually physically in the same building as you, or in some other particular location.
  • finger username --- gives you lots of information about that user, e.g. when they last read their mail and whether they're logged in. Often people put other practical information, such as phone numbers and addresses, in a file called .plan. This information is also displayed by 'finger'.
  • last -1 username --- tells you when the user last logged on and off and from where. Without any options, last will give you a list of everyone's logins.
  • talk username --- lets you have a (typed) conversation with another user
  • write username --- lets you exchange one-line messages with another user
  • elm --- lets you send e-mail messages to people around the world (and, of course, read them). It's not the only mailer you can use, but the one we recommend. See the [elm page][8], and find out about the departmental [ mailing lists][9] (which you can also find in /user/linguistics/helpfile).

About your (electronic) self

  • whoami --- returns your username. Sounds useless, but isn't. You may need to find out who it is who forgot to log out somewhere, and make sure you have logged out.
  • finger & .plan files
    of course you can finger yourself, too. That can be useful e.g. as a quick check whether you got new mail. Try to create a useful .plan file soon. Look at other people's .plan files for ideas. The file needs to be readable for everyone in order to be visible through 'finger'. Do 'chmod a+r .plan' if necessary. You should realize that this information is accessible from anywhere in the world, not just to other people on turing.
  • passwd --- lets you change your password, which you should do regularly (at least once a year). See the [LRB guide][10] and/or look at [help password][11].
  • ps -u yourusername --- lists your processes. Contains lots of information about them, including the process ID, which you need if you have to kill a process. Normally, when you have been kicked out of a dialin session or have otherwise managed to get yourself disconnected abruptly, this list will contain the processes you need to kill. Those may include the shell (tcsh or whatever you're using), and anything you were running, for example emacs or elm. Be careful not to kill your current shell - the one with the number closer to the one of the ps command you're currently running. But if it happens, don't panic. Just try again :) If you're using an X-display you may have to kill some X processes before you can start them again. These will show only when you use ps -efl, because they're root processes.
  • kill PID --- kills (ends) the processes with the ID you gave. This works only for your own processes, of course. Get the ID by using ps. If the process doesn't 'die' properly, use the option -9. But attempt without that option first, because it doesn't give the process a chance to finish possibly important business before dying. You may need to kill processes for example if your modem connection was interrupted and you didn't get logged out properly, which sometimes happens.
  • quota -v --- show what your disk quota is (i.e. how much space you have to store files), how much you're actually using, and in case you've exceeded your quota (which you'll be given an automatic warning about by the system) how much time you have left to sort them out (by deleting or gzipping some, or moving them to your own computer).
  • du filename --- shows the disk usage of the files and directories in filename (without argument the current directory is used). du -s gives only a total.
  • last yourusername --- lists your last logins. Can be a useful memory aid for when you were where, how long you've been working for, and keeping track of your phonebill if you're making a non-local phonecall for dialling in.
du -hsx * | sort -rh | head -10

* du command -h option : display file sizes in human readable format.
* du command -s option : Show total for each argument.
* du command -x option : Skip directories. (if on different file systems)
* sort command -r option : Reverse the result of comparisons.
* sort command -h option : Compare the numbers.
* head command -10 OR -n 10 option : Displays the first 10 lines of the output.

find the largest files & directories on Linux per directory (cd to /path/)

find . -regextype posix-egrep -regex ".*(rb|js)$"

Find all files ending in .rb or .js.

find . -name "*js" -o -name "*rb"

Same as above, not using regular expressions.

find . -regextype posix-ergep -regex ".*(rb|js)$" -exec grep -l matchNameHere {} \;

Find all files ending in .rb or .js, then search those files for 'matchNameHere'. The -l option prints the line number of the occurance. (http://www.devdaily.com/unix/edu/examples/find.shtml)

find ~/some/directory -name "*rb" -exec basename {} \;

Find all files ending in .rb, then print a list of filenames (no paths). Note that the '\;' is mandatory, it tells find where to stop the command execution

# sed
ls *.rb | awk '{print("mv "$1" "$1)}' | sed 's/\.rb/_service\.rb/2' | /bin/sh

Batch rename all files matching a pattern, insert the _service before the file suffix

find /some/path -name "*rb" -o -name "*yml" | xargs grep -sl "some_phrase" | xargs sed -i -e 's/some_phrase/replacement_phrase'

Find all files ending in rb or yml, then grep those files for some_phrase, then replace those instances using sed

cat /some/file.txt | awk '{print "mkdir /path/to/"$0""}' | /bin/sh

Read input from file, pipe to awk to generate a system command, then execute it.

display file with line numbers
cat file -n
display gzipped file with line numbers
zcat file | cat -n
display file from the i. line to j. line (that is j minus i there)
cat file | head -n j | tail -n j-i
remove first line of the file (efficient)
tail -n +2 file > new_file
remove first line of the file (less efficient)
sed 1d file > newfile
count how many lines of a file contain a pattern
cat file | grep pattern | wc -l
grep pattern -c file
select non-matching lines
cat file | grep -v pattern
get the number of the first line where a pattern occur
cat file -n | grep pattern -m 1 | cut -f1 | tr -d ' '
display N lines after, before, around pattern
grep -A N pattern file
grep -B N pattern file
grep -C N pattern file
display N lines around pattern (only first appearance, useful if you know that pattern is unique)
grep -m 1 -C N pattern file
adding together two (or more) files and create a new file from the result
cat file1 file2 > newfile
create new compressed tar archive
tar -zcvf archive.tar.gz dirname
extract from tar archive
tar -xzf archive.tar.gz
view an tar archive
tar tvf archive.tar.gz
extract tar.bz2
tar jxf filename.tar.bz2
add user
sudo adduser <username>
add group
sudo groupadd <groupname>
add existing user to group
sudo usermod -a -G <groupname> <username>
add a user to multiple groups
sudo usermod -a -G <groupname1>,<groupname2>,<groupname3> <username>
view user's group assignemnts
id <username>
groups <username>
make a directory accessible for a group
change group of all files/directories recursively
sudo chgrp -R <groupname> <directory>
add write permissions to group
sudo chmod -R g+w <directory>
set "GID", so that all new files and directories created under are owned by the group
sudo find <directory> -type d -exec chmod 2775 {} \;
randomize lines of a big file memory efficiently
uuid -v 4 -n $(wc -l < lines.txt) | paste - lines.txt | sort | cut -f2 > lines_random.txt
change hostname
sudo vim /etc/hostname -> change old name to new name
sudo vim /etc/hosts -> change old name to new name
sudo /etc/init.d/hostname restart
search a device for bad blocks
sudo badblocks -v <device_name>
sudo badblocks -v /dev/md2
renice (reset priority of processgroup): niceness: 0 (normal) - 19 (low)
sudo renice -n <niceness> -g <pid>
count and filter csv files
remove header (first line), the separator in csv file is ","; count the rows where the value in the second column is above a threshold value (4.5)
tail -n +2 file.csv | awk -F "," '$2 >= <my threshold value>' | wc -l
pg_dump ssh tunnel
ssh -o "Compression=no" mydbserver "pg_dump -Fc -Z9 -U postgres mydb" > mydb.dump
generate random password
of course there are tons of ways to do this
you can replace 32 with the desired password length
< /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c${1:-32};echo;
force the user to change their password upon next login
sudo chage -d 0 <username>
server shutdown
sudo shutdown -h now
sudo poweroff
encrypt/decrypt file with openssl
openssl aes-256-cbc -salt -in <file> -out <encrypted_file> -k <passphrase>
openssl aes-256-cbc -d -in <encrypted_file> -out <file> -k <passphrase>
generate your dhparam.pem file
openssl dhparam -out /etc/nginx/ssl/dhparam.pem 2048
search for pattern in codebase
grep -inIEr --color=ALWAYS "<pattern_to_search>" .
copy directory tree recursively without overwrite
rsync -a -v --ignore-existing <src_dir> <dst_dir>
find processes with most memory usage
ps -e -orss,%mem,cputime,%cpu,pid,args | sort -b -k1,1n | pr -TW$COLUMNS
find processes with maximum file descriptors
lsof -Fpcn | nawk '/^p/ { pid=substr($0,2) } /^c/ { cmd=substr($0,2) } /^n/ { fd[cmd"["pid"]"]++ } END  { for (cc in fd) printf("%-30s %i\n",cc,fd[cc]) } ' | sort -n -k 2 | tail -30

results matching ""

    No results matching ""