Linux Command Line and Shell Scripting Bible

Linux Command Line and Shell Scripting Bible

TableofContents
1. Introduction
1. WhoShouldReadThisBook
2. HowThisBookIsOrganized
3. MinimumRequirements
4. WheretoGofromHere
2. PartI:TheLinuxCommandLine
3. Chapter1:StartingwithLinuxShells
1. WhatIsLinux?
2. LinuxDistributions
3. Summary
4. Chapter2:GettingtotheShell
1. ReachingtheCommandLine
2. AccessingCLIviaaLinuxConsoleTerminal
3. AccessingCLIviaGraphicalTerminalEmulation
4. UsingtheGNOMETerminalEmulator
5. UsingtheKonsoleTerminalEmulator
6. UsingthextermTerminalEmulator
7. Summary
5. Chapter3:BasicbashShellCommands
1. StartingtheShell
2. UsingtheShellPrompt
3. InteractingwiththebashManual
4. NavigatingtheFilesystem
5. ListingFilesandDirectories
6. HandlingFiles
7. ManagingDirectories
8. ViewingFileContents
9. Summary
6. Chapter4:MorebashShellCommands
1. MonitoringPrograms
2. MonitoringDiskSpace
3. WorkingwithDataFiles
4. Summary
7. Chapter5:UnderstandingtheShell
1. ExploringShellTypes
2. ExploringParentandChildShellRelationships
3. UnderstandingShellBuilt-InCommands
4. Summary
8. Chapter6:UsingLinuxEnvironmentVariables
1. ExploringEnvironmentVariables
2. SettingUser-DefinedVariables
3. RemovingEnvironmentVariables
4. UncoveringDefaultShellEnvironmentVariables
5. SettingthePATHEnvironmentVariable
6. LocatingSystemEnvironmentVariables
7. LearningaboutVariableArrays
8. Summary
9. Chapter7:UnderstandingLinuxFilePermissions
1. LinuxSecurity
2. UsingLinuxGroups
3. DecodingFilePermissions
4. ChangingSecuritySettings
5. SharingFiles
6. Summary
10. Chapter8:ManagingFilesystems
1. ExploringLinuxFilesystems
2. WorkingwithFilesystems
3. ManagingLogicalVolumes
4. Summary
11. Chapter9:InstallingSoftware
1. PackageManagementPrimer
2. TheDebian-BasedSystems
3. TheRedHat–BasedSystems
4. InstallingfromSourceCode
5. Summary
12. Chapter10:WorkingwithEditors
1. VisitingthevimEditor
2. NavigatingthenanoEditor
3. ExploringtheemacsEditor
4. ExploringtheKDEFamilyofEditors
5. ExploringtheGNOMEEditor
6. Summary
13. PartII:ShellScriptingBasics
14. Chapter11:BasicScriptBuilding
1. UsingMultipleCommands
2. CreatingaScriptFile
3. DisplayingMessages
4. UsingVariables
5. RedirectingInputandOutput
6. Pipes
7. PerformingMath
8. ExitingtheScript
9. Summary
15. Chapter12:UsingStructuredCommands
1. Workingwiththeif-thenStatement
2. Exploringtheif-then-elseStatement
3. Nestingifs
4. TryingthetestCommand
5. ConsideringCompoundTesting
6. WorkingwithAdvancedif-thenFeatures
7. ConsideringthecaseCommand
8. Summary
16. Chapter13:MoreStructuredCommands
1. TheforCommand
2. TheC-StyleforCommand
3. ThewhileCommand
4. TheuntilCommand
5. NestingLoops
6. LoopingonFileData
7. ControllingtheLoop
8. ProcessingtheOutputofaLoop
9. PracticalExamples
10. Summary
17. Chapter14:HandlingUserInput
1. PassingParameters
2. UsingSpecialParameterVariables
3. BeingShifty
4. WorkingwithOptions
5. StandardizingOptions
6. GettingUserInput
7. Summary
18. Chapter15:PresentingData
1. UnderstandingInputandOutput
2. RedirectingOutputinScripts
3. RedirectingInputinScripts
4. CreatingYourOwnRedirection
5. ListingOpenFileDescriptors
6. SuppressingCommandOutput
7. UsingTemporaryFiles
8. LoggingMessages
9. PracticalExample
10. Summary
19. Chapter16:ScriptControl
1. HandlingSignals
2. RunningScriptsinBackgroundMode
3. RunningScriptswithoutaHang-Up
4. ControllingtheJob
5. BeingNice
6. RunningLikeClockwork
7. Summary
20. PartIII:AdvancedShellScripting
21. Chapter17:CreatingFunctions
1. BasicScriptFunctions
2. ReturningaValue
3. UsingVariablesinFunctions
4. ArrayVariablesandFunctions
5. FunctionRecursion
6. CreatingaLibrary
7. UsingFunctionsontheCommandLine
8. FollowingaPracticalExample
9. Summary
22. Chapter18:WritingScriptsforGraphicalDesktops
1. CreatingTextMenus
2. DoingWindows
3. GettingGraphic
4. Summary
23. Chapter19:Introducingsedandgawk
1. ManipulatingText
2. CommandingatthesedEditorBasics
3. Summary
24. Chapter20:RegularExpressions
1. WhatAreRegularExpressions?
2. DefiningBREPatterns
3. ExtendedRegularExpressions
4. RegularExpressionsinAction
5. Summary
25. Chapter21:Advancedsed
1. LookingatMultilineCommands
2. HoldingSpace
3. NegatingaCommand
4. ChangingtheFlow
5. ReplacingviaaPattern
6. PlacingsedCommandsinScripts
7. CreatingsedUtilities
8. Summary
26. Chapter22:Advancedgawk
1. UsingVariables
2. WorkingwithArrays
3. UsingPatterns
4. StructuredCommands
5. FormattedPrinting
6. Built-InFunctions
7. User-DefinedFunctions
8. WorkingthroughaPracticalExample
9. Summary
27. Chapter23:WorkingwithAlternativeShells
1. WhatIsthedashShell?
2. ThedashShellFeatures
3. Scriptingindash
4. ThezshShell
5. PartsofthezshShell
6. Scriptingwithzsh
7. Summary
28. PartIV:CreatingPracticalScripts
29. Chapter24:WritingSimpleScriptUtilities
1. PerformingArchives
2. ManagingUserAccounts
3. MonitoringDiskSpace
4. Summary
30. Chapter25:ProducingScriptsforDatabase,Web,andE-Mail
1. UsingaMySQLDatabase
2. UsingtheWeb
3. UsingE-Mail
4. Summary
31. Chapter26:CreatingFunLittleShellScripts
1. SendingaMessage
2. ObtainingaQuote
3. GeneratinganExcuse
4. Summary
32. AppendixA:QuickGuidetobashCommands
1. ReviewingBuilt-InCommands
2. LookingatCommonbashCommands
3. AssessingEnvironmentVariables
33. AppendixB:QuickGuidetosedandgawk
1. ThesedEditor
2. ThegawkProgram
34. EndUserLicenseAgreement
ListofIllustrations
1. Figure1.1
2. Figure1.2
3. Figure1.3
4. Figure1.4
5. Figure1.5
6. Figure1.6
7. Figure2.1
8. Figure2.2
9. Figure2.3
10. Figure2.4
11. Figure2.5
12. Figure3.1
13. Figure3.2
14. Figure3.3
15. Figure4.1
16. Figure5.1
17. Figure5.2
18. Figure5.3
19. Figure7.1
20. Figure8.1
21. Figure9.1
22. Figure10.1
23. Figure10.2
24. Figure10.3
25. Figure10.4
26. Figure10.5
27. Figure10.6
28. Figure10.7
29. Figure10.8
30. Figure10.9
31. Figure10.10
32. Figure10.11
33. Figure10.12
34. Figure10.13
35. Figure10.14
36. Figure11.1
37. Figure11.2
38. Figure18.1
39. Figure18.2
40. Figure18.3
41. Figure18.4
42. Figure18.5
43. Figure18.6
44. Figure18.7
45. Figure18.8
46. Figure18.9
47. Figure18.10
48. Figure18.11
49. Figure18.12
50. Figure18.13
51. Figure20.1
52. Figure21.1
53. Figure24.1
54. Figure25.1
55. Figure25.2
ListofTables
1. Table1.1
2. Table1.2
3. Table1.3
4. Table1.4
5. Table1.5
6. Table1.6
7. Table2.1
8. Table2.2
9. Table2.3
10. Table2.4
11. Table2.5
12. Table2.6
13. Table2.7
14. Table2.8
15. Table2.9
16. Table2.10
17. Table2.11
18. Table2.12
19. Table2.13
20. Table2.14
21. Table2.15
22. Table2.16
23. Table2.17
24. Table2.18
25. Table3.1
26. Table3.2
27. Table3.3
28. Table4.1
29. Table4.2
30. Table4.3
31. Table4.4
32. Table4.5
33. Table4.6
34. Table4.7
35. Table4.8
36. Table4.9
37. Table5.1
38. Table6.1
39. Table6.2
40. Table7.1
41. Table7.2
42. Table7.3
43. Table7.4
44. Table7.5
45. Table7.6
46. Table8.1
47. Table8.2
48. Table8.3
49. Table8.4
50. Table8.5
51. Table8.6
52. Table9.1
53. Table9.2
54. Table9.3
55. Table9.4
56. Table9.5
57. Table9.6
58. Table9.7
59. Table10.1
60. Table10.2
61. Table10.3
62. Table10.4
63. Table10.5
64. Table11.1
65. Table11.2
66. Table12.1
67. Table12.2
68. Table12.3
69. Table12.4
70. Table14.1
71. Table15.1
72. Table15.2
73. Table16.1
74. Table16.2
75. Table17.1
76. Table18.1
77. Table18.2
78. Table18.3
79. Table18.4
80. Table19.1
81. Table19.2
82. Table20.1
83. Table21.1
84. Table22.1
85. Table22.2
86. Table22.3
87. Table22.4
88. Table22.5
89. Table22.6
90. Table23.1
91. Table23.2
92. Table23.3
93. Table23.4
94. Table23.5
95. Table25.1
96. Table25.2
97. TableA.1
98. TableA.2
99. TableA.3
100. TableB.1
101. TableB.2
102. TableB.3
103. TableB.4
Introduction
WelcometothethirdeditionofLinuxCommandLineandShellScriptingBible.Likeall
booksintheBible series, you can expect to find both hands-on tutorials and real-world
information, as well as reference and background information that provide a context for
what you are learning. This book is a fairly comprehensive resource on the Linux
command line and shell commands. By the time you have completed Linux Command
LineandShellScriptingBible,youwillbewellpreparedtowriteyourownshellscripts
thatcanautomatepracticallyanytaskonyourLinuxsystem.
WhoShouldReadThisBook
If you’re a system administrator in a Linux environment, you’ll benefit greatly by
knowing how to write shell scripts. The book doesn’t walk you through the process of
settingupaLinuxsystem,butafteryouhaveitrunning,you’llwanttostartautomating
someoftheroutineadministrativetasks.That’swhereshellscriptingcomesin,andthat’s
where this book helps you out. This book demonstrates how to automate any
administrativetaskusingshellscripts,frommonitoringsystemstatisticsanddatafilesto
generatingreportsforyourboss.
If you’re a home Linux enthusiast, you’ll also benefit from Linux Command Line and
ShellScriptingBible. Nowadays, it’s easy to get lost in the graphical world of pre-built
widgets.MostdesktopLinuxdistributionstrytheirbesttohidetheLinuxsystemfromthe
typicaluser.However,sometimesyoumustknowwhat’sgoingonunderthehood.This
bookshowsyouhowtoaccesstheLinuxcommandlinepromptandwhattodowhenyou
get there. Often, performing simple tasks, such as file management, can be done more
quickly from the command line than from a fancy graphical interface. You can use a
wealthofcommandsfromthecommandline,andthisbookshowsyouhowtousethem.
HowThisBookIsOrganized
This book leads you through the basics of the Linux command line and into more
complicatedtopics,suchascreatingyourownshellscripts.Thebookisdividedintofour
parts,eachonebuildingonthepreviousparts.
PartIassumesthatyoueitherhaveaLinuxsystemrunningorarelookingintogettinga
Linuxsystem.Chapter1,“StartingwithLinuxShells,”describesthepartsofatotalLinux
systemandshowshowtheshellfitsin.AfterdescribingthebasicsoftheLinuxsystem,
thispartcontinueswiththefollowing:
Usingaterminalemulationpackagetoaccesstheshell(Chapter2)
Introducingthebasicshellcommands(Chapter3)
Usingmoreadvancedshellcommandstopeekatsysteminformation(Chapter4)
Understandingwhattheshellisusedfor(Chapter5)
Workingwithshellvariablestomanipulatedata(Chapter6)
UnderstandingtheLinuxfilesystemandsecurity(Chapter7)
WorkingwithLinuxfilesystemsfromthecommandline(Chapter8)
Installingandupdatingsoftwarefromthecommandline(Chapter9)
UsingtheLinuxeditorstostartwritingshellscripts(Chapter10)
InPartII,youbeginwritingshellscripts.Asyougothroughthechapters,you’lldothe
following:
Learnhowtocreateandrunshellscripts(Chapter11)
Altertheprogramflowinashellscript(Chapter12)
Iteratethroughcodesections(Chapter13)
Handledatafromtheuserinyourscripts(Chapter14)
SeedifferentmethodsforstoringanddisplayingdatafromyourScript(Chapter15)
Controlhowandwhenyourshellscriptsrunonthesystem(Chapter16)
Part III dives into more advanced areas of shell script programming, including these
things:
Creatingyourownfunctionstouseinallyourscripts(Chapter17)
UtilizingtheLinuxgraphicaldesktopforinteractingwithyourscriptusers(Chapter
18)
UsingadvancedLinuxcommandstofilterandparsedatafiles(Chapter19)
Usingregularexpressionstodefinedata(Chapter20)
Learningadvancedmethodsofmanipulatingdatainyourscripts(Chapter21)
Generatingreportsfromrawdata(Chapter22)
ModifyingyourshellscriptstoruninotherLinuxshells(Chapter23)
Thelastsectionofthebook,PartIV,demonstrateshowtouseshellscriptsinreal-world
environments.Inthispart,youwilllearnthesethings:
Howtoputallthescriptingfeaturestogethertowriteyourownscripts(Chapter24)
Howtostoreandretrievedatausingdatabases,accessdataontheInternet,andsend
e-mailmessages(Chapter25)
WritemoreadvancedshellscriptstointeractonyourLinuxsystem(Chapter26)
Cautions,Tips,andNotes
You will find many different organizational and typographical features throughout this
bookdesignedtohelpyougetthemostoftheinformation.
Caution
Thisinformationisimportantandissetoffinaseparateparagraphwithaspecial
icon.Cautionsprovideinformationaboutthingstowatchoutfor,whethersimply
inconvenientorpotentiallyhazardoustoyourdataorsystems.
Tip
Tipsprovidehelpfuladvicetomakeyourworkeasierandmoreeffective.Tipsmay
suggestasolutiontoaproblemorabetterwaytoaccomplishatask.
Note
Notesprovideadditional,ancillaryinformationthatishelpful,butsomewhatoutside
ofthecurrentpresentationofinformation.
Downloadablecode
Youcanobtainthebook’scodefilesatwww.wiley.com/go/linuxcommandline.
MinimumRequirements
Linux Command Line and Shell Scripting Bible doesn’t focus on any specific Linux
distribution, so you can follow along in the book using any Linux system you have
available. The bulk of the book references the bash shell, which is the default shell for
mostLinuxsystems.
WheretoGofromHere
Afteryou’vefinishedreadingLinuxCommandLineandShellScriptingBible,you’rewell
on your way to incorporating Linux commands in your daily Linux work. In the everchangingworldofLinux,it’salwaysagoodideatostayintouchwithnewdevelopments.
Often,Linuxdistributionschange,addingnewfeaturesandremovingolderones.Tokeep
yourknowledgeofLinuxfresh,alwaysstaywell-informed.FindagoodLinuxforumsite
andmonitorwhat’shappeningintheLinuxworld.ManypopularLinuxnewssites,such
asSlashdotandDistrowatch,provideup-to-the-minuteinformationaboutnewadvancesin
Linux.
PartI
TheLinuxCommandLine
InThisPart
1. Chapter1StartingwithLinuxShells
1. Chapter2GettingtotheShell
1. Chapter3BasicbashShellCommands
1. Chapter4MorebashShellCommands
1. Chapter5UnderstandingtheShell
1. Chapter6UsingLinuxEnvironmentVariables
1. Chapter7UnderstandingLinuxFilePermissions
1. Chapter8ManagingFilesystems
1. Chapter9InstallingSoftware
1. Chapter10WorkingwithEditors
Chapter1
StartingwithLinuxShells
InThisChapter
1. WhatisLinux?
2. PartsoftheLinuxkernel
3. ExploringtheLinuxdesktop
4. VisitingLinuxdistributions
BeforeyoucandiveintoworkingwiththeLinuxcommandlineandshells,you
shouldfirstunderstandwhatLinuxis,whereitcamefrom,andhowitworks.This
chapterwalksyouthroughwhatLinuxisandexplainswheretheshellandcommand
linefitintheoverallLinuxpicture.
WhatIsLinux?
If you’ve never worked with Linux before, you may be confused about why so many
differentversionsareavailable.I’msureyouhavebeenconfusedbyvarioustermssuchas
distribution, LiveCD, and GNU when looking at Linux packages. Wading through the
worldofLinuxforthefirsttimecanbeatrickyexperience.Thischaptertakessomeofthe
mysteryoutoftheLinuxsystembeforeyoustartworkingoncommandsandscripts.
First,fourmainpartsmakeupaLinuxsystem:
TheLinuxkernel
TheGNUutilities
Agraphicaldesktopenvironment
Applicationsoftware
EachofthesepartshasaspecificjobintheLinuxsystem.Nopartisveryusefulbyitself.
Figure1.1showsabasicdiagramofhowthepartsfittogethertocreatetheoverallLinux
system.
Figure1.1TheLinuxsystem
Thissectiondescribesthesefourmainpartsindetailandgivesyouanoverviewofhow
theyworktogethertocreateacompleteLinuxsystem.
LookingintotheLinuxKernel
The core of the Linux system is the kernel. The kernel controls all the hardware and
software on the computer system, allocating hardware when necessary and executing
softwarewhenrequired.
Ifyou’vebeenfollowingtheLinuxworldatall,nodoubtyou’veheardthenameLinus
Torvalds.LinusisthepersonresponsibleforcreatingthefirstLinuxkernelsoftwarewhen
he was a student at the University of Helsinki. He intended it to be a copy of the Unix
system,atthetimeapopularoperatingsystemusedatmanyuniversities.
After developing the Linux kernel, Linus released it to the Internet community and
solicited suggestions for improving it. This simple process started a revolution in the
world of computer operating systems. Soon Linus was receiving suggestions from
studentsaswellasprofessionalprogrammersfromaroundtheworld.
Allowing anyone to change programming code in the kernel would result in complete
chaos.Tosimplifythings,Linusactedasacentralpointforallimprovementsuggestions.
It was ultimately Linus’s decision whether or not to incorporate suggested code in the
kernel.ThissameconceptisstillinplacewiththeLinuxkernelcode,exceptthatinstead
ofjustLinuscontrollingthekernelcode,ateamofdevelopershastakenonthetask.
Thekernelisprimarilyresponsibleforfourmainfunctions:
Systemmemorymanagement
Softwareprogrammanagement
Hardwaremanagement
Filesystemmanagement
Thefollowingsectionsexploreeachofthesefunctionsinmoredetail.
SystemMemoryManagement
Oneoftheprimaryfunctionsoftheoperatingsystemkernelismemorymanagement.Not
onlydoesthekernelmanagethephysicalmemoryavailableontheserver,butitcanalso
createandmanagevirtualmemory,ormemorythatdoesnotactuallyexist.
Itdoesthisbyusingspaceontheharddisk,calledtheswapspace.Thekernelswapsthe
contents of virtual memory locations back and forth from the swap space to the actual
physical memory. This allows the system to think there is more memory available than
whatphysicallyexists,asshowninFigure1.2.
Figure1.2TheLinuxsystemmemorymap
Thememorylocationsaregroupedintoblockscalledpages.Thekernellocateseachpage
ofmemoryeitherinthephysicalmemoryortheswapspace.Thekernelthenmaintainsa
tableofthememorypagesthatindicateswhichpagesareinphysicalmemoryandwhich
pagesareswappedouttodisk.
The kernel keeps track of which memory pages are in use and automatically copies
memory pages that have not been accessed for a period of time to the swap space area
(calledswappingout),evenifthere’sothermemoryavailable.Whenaprogramwantsto
access a memory page that has been swapped out, the kernel must make room for it in
physicalmemorybyswappingoutadifferentmemorypageandswappingintherequired
page from the swap space. Obviously, this process takes time and can slow down a
running process. The process of swapping out memory pages for running applications
continuesforaslongastheLinuxsystemisrunning.
SoftwareProgramManagement
TheLinuxoperatingsystemcallsarunningprogramaprocess.Aprocesscanruninthe
foreground, displaying output on a display, or it can run in the background, behind the
scenes.ThekernelcontrolshowtheLinuxsystemmanagesalltheprocessesrunningon
thesystem.
Thekernelcreatesthefirstprocess,calledtheinitprocess,tostartallotherprocesseson
the system. When the kernel starts, it loads the init process into virtual memory. As the
kernelstartseachadditionalprocess,itgivesitauniqueareainvirtualmemorytostore
thedataandcodethattheprocessuses.
SomeLinuximplementationscontainatableofprocessestostartautomaticallyonbootup.
OnLinuxsystems,thistableisusuallylocatedinthespecialfile/etc/inittabs.
Other systems (such as the popular Ubuntu Linux distribution) utilize the /etc/init.d
folder, which contains scripts for starting and stopping individual applications at boot
time. The scripts are started via entries under the /etc/rcX.d folders, where X is a run
level.
TheLinuxoperatingsystemusesaninitsystemthatutilizesrunlevels.Arunlevelcanbe
used to direct the init process to run only certain types of processes, as defined in the
/etc/inittabsfileorthe /etc/rcX.dfolders.TherearefiveinitrunlevelsintheLinux
operatingsystem.
At run level 1, only the basic system processes are started, along with one console
terminalprocess.Thisiscalledsingle-usermode.Single-usermodeismostoftenusedfor
emergency filesystem maintenance when something is broken. Obviously, in this mode,
onlyoneperson(usuallytheadministrator)canlogintothesystemtomanipulatedata.
The standard init run level is 3. At this run level, most application software, such as
network support software, is started. Another popular run level in Linux is run level 5.
ThisistherunlevelwherethesystemstartsthegraphicalXWindowsoftwareandallows
youtologinusingagraphicaldesktopwindow.
TheLinuxsystemcancontroltheoverallsystemfunctionalitybycontrollingtheinitrun
level.Bychangingtherunlevelfrom3to5,thesystemcanchangefromaconsole-based
systemtoanadvanced,graphicalXWindowsystem.
In Chapter 4, you’ll see how to use the ps command to view the processes currently
runningontheLinuxsystem.
HardwareManagement
Still another responsibility for the kernel is hardware management. Any device that the
Linuxsystemmustcommunicatewithneedsdrivercodeinsertedinsidethekernelcode.
The driver code allows the kernel to pass data back and forth to the device, acting as a
middlemanbetweenapplicationsandthehardware.Twomethodsareusedforinserting
devicedrivercodeintheLinuxkernel:
Driverscompiledinthekernel
Drivermodulesaddedtothekernel
Previously, the only way to insert device driver code was to recompile the kernel. Each
timeyouaddedanewdevicetothesystem,youhadtorecompilethekernelcode.This
process became even more inefficient as Linux kernels supported more hardware.
Fortunately, Linux developers devised a better method to insert driver code into the
runningkernel.
Programmersdevelopedtheconceptofkernelmodulestoallowyoutoinsertdrivercode
intoarunningkernelwithouthavingtorecompilethekernel.Also,akernelmodulecould
be removed from the kernel when the device was finished being used. This greatly
simplifiedandexpandedusinghardwarewithLinux.
The Linux system identifies hardware devices as special files, called devicefiles. There
arethreeclassificationsofdevicefiles:
Character
Block
Network
Character device files are for devices that can only handle data one character at a time.
Most types of modems and terminals are created as character files. Block files are for
devicesthatcanhandledatainlargeblocksatatime,suchasdiskdrives.
Thenetworkfiletypesareusedfordevicesthatusepacketstosendandreceivedata.This
includes network cards and a special loopback device that allows the Linux system to
communicatewithitselfusingcommonnetworkprogrammingprotocols.
Linux creates special files, called nodes, for each device on the system. All
communication with the device is performed through the device node. Each node has a
uniquenumberpairthatidentifiesittotheLinuxkernel.Thenumberpairincludesamajor
and a minor device number. Similar devices are grouped into the same major device
number.Theminordevicenumberisusedtoidentifyaspecificdevicewithinthemajor
devicegroup.
FilesystemManagement
Unlike some other operating systems, the Linux kernel can support different types of
filesystemstoreadandwritedatatoandfromharddrives.Besideshavingoveradozen
filesystems of its own, Linux can read and write to and from filesystems used by other
operatingsystems,suchasMicrosoftWindows.Thekernelmustbecompiledwithsupport
foralltypesoffilesystemsthatthesystemwilluse.Table1.1liststhestandardfilesystems
thataLinuxsystemcanusetoreadandwritedata.
Table1.1LinuxFilesystems
Filesystem
ext
ext2
ext3
ext4
hpfs
jfs
iso9660
minix
msdos
ncp
nfs
Description
LinuxExtendedfilesystem—theoriginalLinuxfilesystem
Secondextendedfilesystem,providedadvancedfeaturesoverext
Thirdextendedfilesystem,supportsjournaling
Fourthextendedfilesystem,supportsadvancedjournaling
OS/2high-performancefilesystem
IBM’sjournalingfilesystem
ISO9660filesystem(CD-ROMs)
MINIXfilesystem
MicrosoftFAT16
Netwarefilesystem
NetworkFileSystem
ntfs
SupportforMicrosoftNTfilesystem
proc
Accesstosysteminformation
ReiserFS AdvancedLinuxfilesystemforbetterperformanceanddiskrecovery
smb
SambaSMBfilesystemfornetworkaccess
sysv
OlderUnixfilesystem
ufs
BSDfilesystem
umsdos
Unix-likefilesystemthatresidesontopofmsdos
vfat
Windows95filesystem(FAT32)
XFS
High-performance64-bitjournalingfilesystem
AnyharddrivethataLinuxserveraccessesmustbeformattedusingoneofthefilesystem
typeslistedinTable1.1.
The Linux kernel interfaces with each filesystem using the Virtual File System (VFS).
This provides a standard interface for the kernel to communicate with any type of
filesystem.VFScachesinformationinmemoryaseachfilesystemismountedandused.
TheGNUUtilities
Besideshavingakerneltocontrolhardwaredevices,acomputeroperatingsystemneeds
utilitiestoperformstandardfunctions,suchascontrollingfilesandprograms.WhileLinus
created the Linux system kernel, he had no system utilities to run on it. Fortunately for
him,atthesametimehewasworking,agroupofpeoplewereworkingtogetheronthe
Internet trying to develop a standard set of computer system utilities that mimicked the
popularUnixoperatingsystem.
The GNU organization (GNU stands for GNU’s Not Unix) developed a complete set of
Unix utilities, but had no kernel system to run them on. These utilities were developed
underasoftwarephilosophycalledopensourcesoftware(OSS).
The concept of OSS allows programmers to develop software and then release it to the
world with no licensing fees attached. Anyone can use the software, modify it, or
incorporate it into his or her own system without having to pay a license fee. Uniting
Linus’s Linux kernel with the GNU operating system utilities created a complete,
functional,freeoperatingsystem.
WhilethebundlingoftheLinuxkernelandGNUutilitiesisoftenjustcalledLinux,you
will see some Linux purists on the Internet refer to it as the GNU/Linux system to give
credittotheGNUorganizationforitscontributionstothecause.
TheCoreGNUUtilities
TheGNUprojectwasmainlydesignedforUnixsystemadministratorstohaveaUnix-like
environment available. This focus resulted in the project porting many common Unix
systemcommandlineutilities.ThecorebundleofutilitiessuppliedforLinuxsystemsis
calledthecoreutilspackage.
TheGNUcoreutilspackageconsistsofthreeparts:
Utilitiesforhandlingfiles
Utilitiesformanipulatingtext
Utilitiesformanagingprocesses
Each of these three main groups of utilities contains several utility programs that are
invaluabletotheLinuxsystemadministratorandprogrammer.Thisbookcoverseachof
theutilitiescontainedintheGNUcoreutilspackageindetail.
TheShell
The GNU/Linux shell is a special interactive utility. It provides a way for users to start
programs, manage files on the filesystem, and manage processes running on the Linux
system. The core of the shell is the command prompt. The command prompt is the
interactivepartoftheshell.Itallowsyoutoentertextcommands,andthenitinterpretsthe
commandsandexecutestheminthekernel.
The shell contains a set of internal commands that you use to control things such as
copyingfiles,movingfiles,renamingfiles,displayingtheprogramscurrentlyrunningon
thesystem,andstoppingprogramsrunningonthesystem.Besidestheinternalcommands,
the shell also allows you to enter the name of a program at the command prompt. The
shellpassestheprogramnameofftothekerneltostartit.
You can also group shell commands into files to execute as a program. Those files are
called shellscripts. Any command that you can execute from the command line can be
placedinashellscriptandrunasagroupofcommands.Thisprovidesgreatflexibilityin
creating utilities for commonly run commands, or processes that require several
commandsgroupedtogether.
There are quite a few Linux shells available to use on a Linux system. Different shells
havedifferentcharacteristics,somebeingmoreusefulforcreatingscriptsandsomebeing
moreusefulformanagingprocesses.ThedefaultshellusedinallLinuxdistributionsisthe
bash shell. The bash shell was developed by the GNU project as a replacement for the
standard Unix shell, called the Bourne shell (after its creator). The bash shell name is a
playonthiswording,referredtoasthe“Bourneagainshell.”
Inadditiontothebashshell,wewillcoverseveralotherpopularshellsinthisbook.Table
1.2liststhedifferentshellswewillexamine.
Table1.2LinuxShells
Shell
Description
Asimple,lightweightshellthatrunsinlow-memoryenvironmentsbuthasfull
ash
compatibilitywiththebashshell
AprogrammingshellcompatiblewiththeBourneshellbutsupportingadvanced
korn
programmingfeatureslikeassociativearraysandfloating-pointarithmetic
AshellthatincorporateselementsfromtheCprogramminglanguageintoshell
tcsh
scripts
Anadvancedshellthatincorporatesfeaturesfrombash,tcsh,andkorn,providing
zsh
advancedprogrammingfeatures,sharedhistoryfiles,andthemedprompts
MostLinuxdistributionsincludemorethanoneshell,althoughusuallytheypickoneof
them to be the default. If your Linux distribution includes multiple shells, feel free to
experimentwithdifferentshellsandseewhichonefitsyourneeds.
TheLinuxDesktopEnvironment
In the early days of Linux (the early 1990s) all that was available was a simple text
interfacetotheLinuxoperatingsystem.Thistextinterfaceallowedadministratorstostart
programs,controlprogramoperations,andmovefilesaroundonthesystem.
With the popularity of Microsoft Windows, computer users expected more than the old
textinterfacetoworkwith.ThisspurredmoredevelopmentintheOSScommunity,and
theLinuxgraphicaldesktopsemerged.
Linux is famous for being able to do things in more than one way, and no place is this
morerelevantthaningraphicaldesktops.Thereareaplethoraofgraphicaldesktopsyou
can choose from in Linux. The following sections describe a few of the more popular
ones.
TheXWindowSystem
Twobasicelementscontrolyourvideoenvironment:thevideocardinyourPCandyour
monitor.Todisplayfancygraphicsonyourcomputer,theLinuxsoftwareneedstoknow
how to talk to both of them. The X Window software is the core element in presenting
graphics.
The X Window software is a low-level program that works directly with the video card
andmonitorinthePC,anditcontrolshowLinuxapplicationscanpresentfancywindows
andgraphicsonyourcomputer.
Linuxisn’ttheonlyoperatingsystemthatusesXWindow;versionsarewrittenformany
different operating systems. In the Linux world, several different software packages can
implementit.
ThemostpopularpackageisX.org.Itprovidesanopensourcesoftwareimplementation
oftheXWindowsystemandsupportsmanyofthenewervideocardsusedtoday.
TwootherXWindowpackagesaregaininginpopularity.TheFedoraLinuxdistributionis
experimenting with the Wayland software, and the Ubuntu Linux distribution has
developedtheMirdisplayserverforusewithitsdesktopenvironment.
When you first install a Linux distribution, it attempts to detect your video card and
monitor, and then it creates an X Window configuration file that contains the required
information. During installation, you may notice a time when the installation program
scansyourmonitorforsupportedvideomodes.Sometimes,thiscausesyourmonitortogo
blank for a few seconds. Because there are lots of different types of video cards and
monitors,thisprocesscantakeawhiletocomplete.
ThecoreXWindowsoftwareproducesagraphicaldisplayenvironment,butnothingelse.
Although this is fine for running individual applications, it is not useful for day-to-day
computer use. No desktop environment allows users to manipulate files or launch
programs. To do that, you need a desktop environment on top of the X Window system
software.
TheKDEDesktop
TheKDesktopEnvironment(KDE)wasfirstreleasedin1996asanopensourceproject
toproduceagraphicaldesktopsimilartotheMicrosoftWindowsenvironment.TheKDE
desktopincorporatesallthefeaturesyouareprobablyfamiliarwithifyouareaWindows
user. Figure 1.3 shows a sample KDE 4 desktop running in the openSUSE Linux
distribution.
Figure1.3TheKDE4desktoponanopenSUSELinuxsystem
TheKDEdesktopallowsyoutoplacebothapplicationandfileiconsinaspecialareaon
the desktop. If you click an application icon, the Linux system starts the application. If
youclickafileicon,theKDEdesktopattemptstodeterminewhatapplicationtostartto
handlethefile.
ThebaratthebottomofthedesktopiscalledthePanel.ThePanelconsistsoffourparts:
TheKmenu:MuchliketheWindowsStartmenu,theKmenucontainslinkstostart
installedapplications.
Programshortcuts:Thesearequicklinkstostartapplicationsdirectlyfromthe
Panel.
Thetaskbar:Thetaskbarshowsiconsforapplicationscurrentlyrunningonthe
desktop.
Applets:ThesearesmallapplicationsthathaveaniconinthePanelthatoftencan
changedependingoninformationfromtheapplication.
The Panel features are similar to what you would find in Windows. In addition to the
desktopfeatures,theKDEprojecthasproducedawideassortmentofapplicationsthatrun
intheKDEenvironment.
TheGNOMEDesktop
The GNU Network Object Model Environment (GNOME) is another popular Linux
desktop environment. First released in 1999, GNOME has become the default desktop
environmentformanyLinuxdistributions.(However,themostpopularisRedHatLinux.)
AlthoughGNOMEchosetodepartfromthestandardMicrosoftWindowslook-and-feel,it
incorporatesmanyfeaturesthatmostWindowsusersarecomfortablewith:
Adesktopareaforicons
Apanelareaforshowingrunningapplications
Drag-and-dropcapabilities
Figure1.4showsthestandardGNOMEdesktopusedintheCentOSLinuxdistribution.
Figure1.4AGNOMEdesktoponaCentOSLinuxsystem
Not to be outdone by KDE, the GNOME developers have also produced a host of
graphicalapplicationsthatintegratewiththeGNOMEdesktop.
TheUnityDesktop
If you’re using the Ubuntu Linux distribution, you’ll notice that it’s somewhat different
from both the KDE and GNOME desktop environments. Canonical, the company
responsible for developing Ubuntu, has decided to embark on its own Linux desktop
environment,calledUnity.
The Unity desktop gets its name from the goal of the project — to provide a single
desktop experience for workstations, tablet devices, and mobile devices. The Unity
desktop works the same whether you’re running Ubuntu on a workstation or a mobile
phone!Figure1.5showsanexampleoftheUnitydesktopinUbuntu14.04LTS.
Figure1.5TheUnitydesktopontheUbuntuLinuxdistribution
OtherDesktops
The downside to a graphical desktop environment is that it requires a fair amount of
system resources to operate properly. In the early days of Linux, a hallmark and selling
feature of Linux was its ability to operate on older, less powerful PCs that the newer
Microsoft desktop products couldn’t run on. However, with the popularity of KDE and
GNOMEdesktops,thishaschanged,becauseittakesjustasmuchmemorytorunaKDE
orGNOMEdesktopasthelatestMicrosoftdesktopenvironment.
If you have an older PC, don’t be discouraged. The Linux developers have banded
together to take Linux back to its roots. They’ve created several low-memory–oriented
graphicaldesktopapplicationsthatprovidebasicfeaturesthatrunperfectlyfineonolder
PCs.
Althoughthesegraphicaldesktopsdon’thaveaplethoraofapplicationsdesignedaround
them,theystillrunmanybasicgraphicalapplicationsthatsupportfeaturessuchasword
processing,spreadsheets,databases,drawing,and,ofcourse,multimediasupport.
Table1.3 shows some of the smaller Linux graphical desktop environments that can be
usedonlower-poweredPCsandlaptops.
Table1.3OtherLinuxGraphicalDesktops
Desktop
Fluxbox
Description
Abare-bonesdesktopthatdoesn’tincludeaPanel,onlyapop-upmenuto
launchapplications
Xfce
Adesktopthat’ssimilartotheKDEdesktop,butwithfewergraphicsforlowmemoryenvironments
Joe’sWindowManager,averylightweightdesktopidealforlow-memoryand
JWM
low-diskspaceenvironments
SupportssomeadvanceddesktopfeaturessuchasvirtualdesktopsandPanels,
Fvwm
butrunsinlow-memoryenvironments
fvwm95
Derivedfromfvwm,butmadetolooklikeaWindows95desktop
These graphical desktop environments are not as fancy as the KDE and GNOME
desktops,buttheyprovidebasicgraphicalfunctionalityjustfine.Figure1.6showswhat
theJWMdesktopusedinthePuppyLinuxantiXdistributionlookslike.
Figure1.6TheJWMdesktopasseeninthePuppyLinuxdistribution
IfyouareusinganolderPC,tryaLinuxdistributionthatusesoneofthesedesktopsand
seewhathappens.Youmaybepleasantlysurprised.
LinuxDistributions
NowthatyouhaveseenthefourmaincomponentsrequiredforacompleteLinuxsystem,
youmaybewonderinghowyouaregoingtogetthemallputtogethertomakeaLinux
system.Fortunately,otherpeoplehavealreadydonethatforyou.
A complete Linux system package is called a distribution. Many different Linux
distributionsareavailabletomeetjustaboutanycomputingrequirementyoucouldhave.
Most distributions are customized for a specific user group, such as business users,
multimedia enthusiasts, software developers, or average home users. Each customized
distributionincludesthesoftwarepackagesrequiredtosupportspecializedfunctions,such
as audio- and video-editing software for multimedia enthusiasts, or compilers and
integrateddevelopmentenvironments(IDEs)forsoftwaredevelopers.
ThedifferentLinuxdistributionsareoftendividedintothreecategories:
FullcoreLinuxdistributions
Specializeddistributions
LiveCDtestdistributions
The following sections describe these different types of Linux distributions and show
someexamplesofLinuxdistributionsineachcategory.
CoreLinuxDistributions
AcoreLinuxdistributioncontainsakernel,oneormoregraphicaldesktopenvironments,
and just about every Linux application that is available, precompiled for the kernel. It
providesone-stopshoppingforacompleteLinuxinstallation.Table1.4showssomeofthe
morepopularcoreLinuxdistributions.
Table1.4CoreLinuxDistributions
Distribution
Slackware
RedHat
Fedora
Gentoo
openSUSE
Debian
Description
OneoftheoriginalLinuxdistributionsets,popularwithLinuxgeeks
AcommercialbusinessdistributionusedmainlyforInternetservers
Aspin-offfromRedHatbutdesignedforhomeuse
AdistributiondesignedforadvancedLinuxusers,containingonlyLinux
sourcecode
Differentdistributionsforbusinessandhomeuse
PopularwithLinuxexpertsandcommercialLinuxproducts
IntheearlydaysofLinux,adistributionwasreleasedasasetoffloppydisks.Youhadto
downloadgroupsoffilesandthencopythemontodisks.Itwouldusuallytake20ormore
diskstomakeanentiredistribution!Needlesstosay,thiswasapainfulexperience.
Nowadays,withhomecomputerscommonlyhavingCDandDVDplayersbuiltin,Linux
distributionsarereleasedaseitheraCDsetorasingleDVD.ThismakesinstallingLinux
mucheasier.
However,beginnersstilloftenrunintoproblemswhentheyinstalloneofthecoreLinux
distributions. To cover just about any situation in which someone might want to use
Linux, a single distribution must include lots of application software. They include
everything from high-end Internet database servers to common games. Because of the
quantity of applications available for Linux, a complete distribution often takes four or
moreCDs.
AlthoughhavinglotsofoptionsavailableinadistributionisgreatforLinuxgeeks,itcan
become a nightmare for beginning Linux users. Most distributions ask a series of
questions during the installation process to determine which applications to load by
default, what hardware is connected to the PC, and how to configure the hardware.
Beginnersoftenfindthesequestionsconfusing.Asaresult,theyofteneitherloadwaytoo
many programs on their computer or don’t load enough and later discover that their
computerwon’tdowhattheywantitto.
Fortunatelyforbeginners,there’samuchsimplerwaytoinstallLinux.
SpecializedLinuxDistributions
AnewsubgroupofLinuxdistributionshasstartedtoappear.Thesearetypicallybasedon
one of the main distributions but contain only a subset of applications that would make
senseforaspecificareaofuse.
In addition to providing specialized software (such as only office products for business
users), customized Linux distributions also attempt to help beginning Linux users by
autodetecting and autoconfiguring common hardware devices. This makes installing
Linuxamuchmoreenjoyableprocess.
Table 1.5 shows some of the specialized Linux distributions available and what they
specializein.
Table1.5SpecializedLinuxDistributions
Distribution
Description
CentOS AfreedistributionbuiltfromtheRedHatEnterpriseLinuxsourcecode
Ubuntu
Afreedistributionforschoolandhomeuse
PCLinuxOS
Afreedistributionforhomeandofficeuse
Mint
Afreedistributionforhomeentertainmentuse
dyne:bolic
AfreedistributiondesignedforaudioandMIDIapplications
PuppyLinux
AfreesmalldistributionthatrunswellonolderPCs
That’s just a small sampling of specialized Linux distributions. There are literally
hundredsofspecializedLinuxdistributions,andmorearepoppingupallthetimeonthe
Internet.Nomatterwhatyourspecialty,you’llprobablyfindaLinuxdistributionmadefor
you.
Many of the specialized Linux distributions are based on the Debian Linux distribution.
TheyusethesameinstallationfilesasDebianbutpackageonlyasmallfractionofafullblownDebiansystem.
TheLinuxLiveCD
ArelativelynewphenomenonintheLinuxworldisthebootableLinuxCDdistribution.
ThisletsyouseewhataLinuxsystemislikewithoutactuallyinstallingit.Mostmodern
PCs can boot from a CD instead of the standard hard drive. To take advantage of this,
some Linux distributions create a bootable CD that contains a sample Linux system
(calledaLinuxLiveCD).BecauseofthelimitationsofthesingleCDsize,thesamplecan’t
containacompleteLinuxsystem,butyou’dbesurprisedatallthesoftwaretheycancram
inthere.TheresultisthatyoucanbootyourPCfromtheCDandrunaLinuxdistribution
withouthavingtoinstallanythingonyourharddrive!
ThisisanexcellentwaytotestvariousLinuxdistributionswithouthavingtomesswith
yourPC.JustpopinaCDandboot!AlltheLinuxsoftwarewillrundirectlyfromtheCD.
You can download lots of Linux LiveCDs from the Internet and burn onto a CD to test
drive.
Table1.6showssomepopularLinuxLiveCDsthatareavailable.
Table1.6LinuxLiveCDDistributions
Distribution
Description
Knoppix
AGermanLinux,thefirstLinuxLiveCDdeveloped
PCLinuxOS
Full-blownLinuxdistributiononaLiveCD
Ubuntu AworldwideLinuxproject,designedformanylanguages
Slax
AliveLinuxCDbasedonSlackwareLinux
PuppyLinux
Afull-featuredLinuxdesignedforolderPCs
Youmaynoticeafamiliarityinthistable.ManyspecializedLinuxdistributionsalsohave
aLinuxLiveCDversion.SomeLinuxLiveCDdistributions,suchasUbuntu,allowyouto
installtheLinuxdistributiondirectlyfromtheLiveCD.Thisenablesyoutobootwiththe
CD,testdrivetheLinuxdistribution,andthenifyoulikeit,installitonyourharddrive.
Thisfeatureisextremelyhandyanduser-friendly.
As with all good things, Linux LiveCDs have a few drawbacks. Because you access
everything from the CD, applications run more slowly, especially if you’re using older,
slowercomputersandCDdrives.Also,becauseyoucan’twritetotheCD,anychanges
youmaketotheLinuxsystemwillbegonethenexttimeyoureboot.
ButadvancesarebeingmadeintheLinuxLiveCDworldthatwillhelptosolvesomeof
theseproblems.Theseadvancesincludetheabilityto:
CopyLinuxsystemfilesfromtheCDtomemory
Copysystemfilestoafileontheharddrive
StoresystemsettingsonaUSBmemorystick
StoreusersettingsonaUSBmemorystick
Some Linux LiveCDs, such as Puppy Linux, are designed with a minimum number of
Linux system files. The LiveCD boot scripts copy them directly into memory when the
CDboots.ThisallowsyoutoremovetheCDfromthecomputerassoonasLinuxboots.
Notonlydoesthismakeyourapplicationsrunmuchfaster(becauseapplicationsrunfaster
frommemory),butitalsogivesyouafreeCDtraytouseforrippingaudioCDsorplaying
videoDVDsfromthesoftwareincludedinPuppyLinux.
OtherLinuxLiveCDsuseanalternativemethodthatallowsyoutoremovetheCDfrom
the tray after booting. It involves copying the core Linux files onto the Windows hard
driveasasinglefile.AftertheCDboots,itlooksforthatfileandreadsthesystemfiles
from it. The dyne:bolic Linux LiveCD uses this technique, which is called docking. Of
course,youmustcopythesystemfiletoyourharddrivebeforeyoucanbootfromtheCD.
A very popular technique for storing data from a live Linux CD session is to use a
commonUSBmemorystick(alsocalledaflashdriveorathumbdrive).Justaboutevery
Linux LiveCD can recognize a plugged-in USB memory stick (even if the stick is
formattedforWindows)andreadandwritefilestoandfromit.Thisallowsyoutoboota
LinuxLiveCD,usetheLinuxapplicationstocreatefiles,storethosefilesonyourmemory
stick, and then access them from your Windows applications later (or from a different
computer).Howcoolisthat?
Summary
ThischapterdiscussedtheLinuxsystemandthebasicsofhowitworks.TheLinuxkernel
is the core of the system, controlling how memory, programs, and hardware all interact
withoneanother.TheGNUutilitiesarealsoanimportantpieceintheLinuxsystem.The
Linuxshell,whichisthemainfocusofthisbook,ispartoftheGNUcoreutilities.The
chapteralsodiscussedthefinalpieceofaLinuxsystem,theLinuxdesktopenvironment.
Things have changed over the years, and Linux now supports several graphical desktop
environments.
ThechapteralsodiscussedthevariousLinuxdistributions.ALinuxdistributionbundles
the various parts of a Linux system into a simple package that you can easily install on
your PC. The Linux distribution world consists of full-blown Linux distributions that
includejustabouteveryapplicationimaginable,aswellasspecializedLinuxdistributions
thatincludeapplicationsfocusedonlyonaspecialfunction.TheLinuxLiveCDcrazehas
created another group of Linux distributions that allow you to easily test-drive Linux
withoutevenhavingtoinstallitonyourharddrive.
In the next chapter, you look at what you need to start your command line and shell
scriptingexperience.You’llseewhatyouneedtodotogettotheLinuxshellutilityfrom
yourfancygraphicaldesktopenvironment.Thesedays,that’snotalwaysaneasything.
Chapter2
GettingtotheShell
InThisChapter
1. Accessingthecommandline
2. ReachingCLIviaaLinuxconsoleterminal
3. ReachingCLIviaagraphicalterminalemulator
4. UsingtheGNOMEterminalemulator
5. UsingtheKonsoleterminalemulator
6. Usingthextermterminalemulator
IntheolddaysofLinux,allyouhadtoworkwithwastheshell.System
administrators,programmers,andsystemusersallsatatsomethingcalledaLinux
consoleterminalenteringshellcommandsandviewingtextoutput.Thesedays,with
graphicaldesktopenvironments,it’sgettinghardertofindashellpromptonthe
systeminordertoentershellcommands.Thischapterdiscusseswhatisrequiredto
reachacommandlineenvironment.Itwalksyouthroughtheterminalemulation
packagesthatyoumayrunintointhevariousLinuxdistributions.
ReachingtheCommandLine
Beforethedaysofgraphicaldesktops,theonlywaytointeractwithaUnixsystemwas
throughatextcommandlineinterface(CLI)providedbytheshell.TheCLIallowedtext
inputonlyandcoulddisplayonlytextandrudimentarygraphicsoutput.
Becauseoftheserestrictions,outputdeviceswerenotveryfancy.Often,youneededonly
a simple dumb terminal to interact with the Unix system. A dumb terminal was usually
nothing more than a monitor and keyboard connected to the Unix system via a
communication cable (usually a multi-wire serial cable). This simple combination
providedaneasywaytoentertextdataintotheUnixsystemandviewtextresults.
As you well know, things are significantly different in today’s Linux environment. Just
about every Linux distribution uses some type of graphical desktop environment.
However,toentershellcommands,youstillneedatextdisplaytoaccesstheshell’sCLI.
The problem now is getting to one. Sometimes finding a way to get a CLI in a Linux
distributionisnotaneasytask.
ConsoleTerminals
OnewaytogettoaCLIistotaketheLinuxsystemoutofgraphicaldesktopmodeand
placeitintextmode.ThisprovidesnothingmorethanasimpleshellCLIonthemonitor,
justlikethedaysbeforegraphicaldesktops.ThismodeiscalledtheLinuxconsolebecause
itemulatestheolddaysofahard-wiredconsoleterminalandisadirectinterfacetothe
Linuxsystem.
WhentheLinuxsystemstarts,itautomaticallycreatesseveralvirtualconsoles.Avirtual
consoleisaterminalsessionthatrunsinLinuxsystemmemory.Insteadofhavingseveral
dumbterminalsconnectedtothecomputer,mostLinuxdistributionsstartfiveorsix(or
sometimes even more) virtual consoles that you can access from a single computer
keyboardandmonitor.
GraphicalTerminals
Thealternativetousingavirtualconsoleterminalistouseaterminalemulationpackage
from within the Linux graphical desktop environment. A terminal emulation package
simulatesworkingonaconsoleterminal,butwithinadesktopgraphicalwindow.Figure
2.1 shows an example of a terminal emulator running in a Linux graphical desktop
environment.
Figure2.1AsimpleterminalemulatorrunningonaLinuxdesktop
Graphical terminal emulation is responsible only for a portion of the Linux graphical
experience.Asawhole,theexperienceisaccomplishedviaseveralcomponents,including
graphical terminal emulation software (called a client). Table 2.1 shows the different
componentsintheLinuxgraphicaldesktopenvironment.
Table2.1GraphicalInterfaceElements
Name
Client
Examples
Graphicalterminalemulator,
desktopenvironment,network
browser
Display
Server
Mir,WaylandCompositor,
Xserver
Window
Manager
Compiz,Metacity,Kwin
Widgets
Library
Athena(Xaw),XIntrinsics
Description
Anapplicationthatrequestsgraphical
services
Elementthatmanagesthedisplay(screen)and
theinputdevices(keyboard,mouse,touch
screen)
Elementthataddsborderstowindowsand
providesfeaturestomoveandmanage
windows
Elementthataddsmenusandappearance
itemsfordesktopenvironmentclients
For dealing with the command line from the desktop, the focus is on the graphical
terminalemulator.YoucanthinkofgraphicalterminalemulatorsasCLIterminals“inthe
GUI”andvirtualconsoleterminalsasCLIterminals“outsidetheGUI.”Understandingthe
variousterminalsandtheirfeaturescanenhanceyourcommandlineexperience.
AccessingCLIviaaLinuxConsoleTerminal
IntheearlydaysofLinux,whenyoubootedupyoursystemyouwouldseealoginprompt
onyourmonitor,andthat’sall.Asmentionedearlier,thisiscalledtheLinuxconsole.It
wastheonlyplaceyoucouldentercommandsforthesystem.
Even though several virtual consoles are created at boot time, many Linux distributions
switch to a graphical environment after the boot sequence completes. This provides the
user with a graphical login and desktop experience. Therefore, in this case, accessing a
virtualconsoleisdonemanually.
In most Linux distributions, you can access one of the Linux virtual consoles using a
simplekeystrokecombination.Usually,youmustholddowntheCtrl+Altkeycombination
and then press a function key (F1 through F7) for the virtual console you want to use.
Function key F2 produces virtual console 2, key F3 produces virtual console 3, key F4
producesvirtualconsole4,andsoon.
Note
LinuxdistributionstypicallyusetheCtrl+AltkeycombinationwitheitherF1orF7to
reachthegraphicalinterface.UbuntuusesF7,whileRHELusesF1.Itisbesttotest
andseewhereyourdistributionputsthegraphicalinterface.
Text mode virtual consoles use the whole screen and start with the text login screen
displayed.AnexampleofatextloginscreenfromavirtualconsoleisshowninFigure2.2.
Figure2.2Linuxvirtualconsoleloginscreen
NoticeinFigure2.2thewordstty2attheendofthefirsttextline.The2intty2indicates
thatitisvirtualconsole2andwasreachedbypressingtheCtrl+Alt+F2keysequence.tty
stands for teletypewriter. Teletypewriter is an old term, indicating a machine used for
sendingmessages.
Note
NotallLinuxdistributionsshowthevirtualconsole’sttynumberattheloginscreen.
You log into a console terminal by entering your user ID after the login: prompt and
typingyourpasswordafterthe Password:prompt.Ifyouhaveneverloggedinthisway
before, be aware that typing your password is a different experience than in a graphical
environment. In a graphical environment, you may see dots or asterisks indicating the
password characters as you type. However, at the virtual console, nothing is displayed
whenyoutypeyourpassword.
Afterloggingintoavirtualconsole,youaretakentotheLinuxCLI.Keepinmindthat,
within the Linux virtual console, you do not have the ability to run any graphical
programs.
Afteryouhaveloggedintoavirtualconsole,youcankeepitactiveandswitchtoanother
virtualconsolewithoutlosingyouractivesession.Youcanswitchbetweenallthevirtual
consoles, with multiple active sessions running. This feature provides a great deal of
flexibilitywhileyouworkattheCLI.
Additionalflexibilitydealswiththevirtualconsole’sappearance.Eventhoughitisatext
modeconsoleterminal,youcanmodifythetextandbackgroundcolors.
Forexample,itmaybeeasieronyoureyestosetthebackgroundoftheterminaltowhite
andthetexttoblack.Afteryouhaveloggedin,youcanaccomplishthismodificationina
coupleofways.Onewayistotypeinthecommandsetterm-inversescreenonandpress
theEnterkey,asshowninFigure2.3.Noticeinthefigurethatthe inversescreenfeature
isbeingturnedonusingtheoptionon.Youcanalsoturnitoffusingtheoffoption.
Figure2.3Linuxvirtualconsolewithinversescreenbeingturnedon
Another way is to type two commands, one after the other. Type setterm-background
white and press Enter, and then type setterm -foreground black and press Enter. Be
carefulbecause,whenyouchangeyourterminalbackgroundfirst,itmaybehardtosee
thecommandsyouaretyping.
Withthecommandsintheprecedingparagraph,youarenotturningfeaturesonandoff,as
with inversescreen.Instead,youhaveachoiceofeightcolors.Thechoicesare black,
red, green, yellow, blue, magenta, cyan, and white (which looks gray on some
distributions). You can get rather creative with your plain text mode console terminals.
Table2.2 shows some options you can use with the settermcommandtohelpimprove
yourconsoleterminal’sreadabilityorappearance.
Table2.2settermOptionsforForegroundandBackgroundAppearance
Option
ParameterChoices
Description
-background
black,red,green,
yellow,blue,magenta,
cyan,orwhite
Changestheterminal’sbackgroundcolorto
theonespecified
-foreground
black,red,green,
yellow,blue,magenta,
cyan,orwhite
Changestheterminal’sforegroundcolor,
specificallytext,totheonespecified
inversescreen
onoroff
-reset
None
-store
None
Switchesthebackgroundcolortothe
foregroundcolorandtheforegroundcolor
tothebackgroundcolor
Changestheterminalappearancebacktoits
defaultsettingandclearsthescreen
Setsthecurrentterminal’sforegroundand
backgroundcolorsasthevaluestobeused
for-reset
Virtual console terminals are great for accessing the CLI outside the GUI. However,
sometimes, you need to access the CLI and run graphical programs. Using a terminal
emulationpackagesolvesthisproblemandisapopularwaytoaccesstheshellCLIfrom
withintheGUI.Thefollowingsectionsdescribecommonsoftwarepackagesthatprovide
graphicalterminalemulation.
AccessingCLIviaGraphicalTerminalEmulation
The graphical desktop environment offers a great deal more variety for CLI access than
the virtual console terminal does. Many graphical terminal emulator packages are
available for the graphical environment. Each package provides its own unique set of
features and options. Some popular graphical terminal emulator packages are shown in
Table2.3alongwiththeirwebsites.
Table2.3PopularGraphicalTerminalEmulatorPackages
Name
Website
Eterm
http://www.eterm.org
FinalTerm
http://finalterm.org
GNOMETerminal
https://help.gnome.org/users/gnome-terminal/stable
Guake
https://github.com/Guake/guake
KonsoleTerminal
http://konsole.kde.org
LillyTerm
http://lilyterm.luna.com.tw/index.html
LXTerminal
http://wiki.lxde.org/en/LXTerminal
mrxvt
https://code.google.com/p/mrxvt
ROXTerm
http://roxterm.sourceforge.net
rxvt
http://sourceforge.net/projects/rxvt
rxvt-unicode
http://software.schmorp.de/pkg/rxvt-unicode
Sakura
https://launchpad.net/sakura
st
http://st.suckless.org
Terminator
https://launchpad.net/terminator
Terminology
http://www.enlightenment.org/p.php?p=about/terminology
tilda
http://tilda.sourceforge.net/tildaabout.php
UXterm
http://manpages.ubuntu.com/manpages/gutsy/man1/uxterm.1.html
Wterm
http://sourceforge.net/projects/wterm
xterm
http://invisible-island.net/xterm
Xfce4Terminal
http://docs.xfce.org/apps/terminal/start
Yakuake
http://extragear.kde.org/apps/yakuake
Although many graphical terminal emulator packages are available, the focus in this
chapterisonthreecommonlyusedones.OfteninstalledinLinuxdistributionsbydefault,
theyareGNOMETerminal,KonsoleTerminal,andxterm.
UsingtheGNOMETerminalEmulator
GNOMETerminalistheGNOMEdesktopenvironment’sdefaultterminalemulator.Many
distributions,suchasRHEL,Fedora,andCentOS,usetheGNOMEdesktopenvironment
by default, and therefore use GNOME Terminal by default. However, other desktop
environments, such as Ubuntu Unity, also use the GNOME terminal as their default
terminal emulator package. It is fairly easy to use and a good terminal emulator for
individuals who are new to Linux. This chapter section walks you through the various
partsofaccessing,configuringandusingtheGNOMEterminalemulator.
AccessingtheGNOMETerminal
Each graphical desktop environment has different methods for accessing the GNOME
terminalemulator.ThissectionlooksataccessingtheGNOMETerminalintheGNOME,
Unity,andKDEdesktopenvironments.
Note
IfyouareusingadifferentdesktopenvironmentthantheoneslistedinTable2.3,you
mustlookthroughthevariousmenusofferedinyourenvironmenttofindthe
GNOMEterminalemulator.Inthemenus,itistypicallynamedTerminal
In the GNOME desktop environment, accessing the GNOME Terminal is fairly
straightforward. From the menu system in the upper-left corner of the window, click
Applications, then select System Tools from the drop-down menu, and finally click
Terminal. Written in shorthand, the directions look like the following:
Applications⇨SystemTools⇨Terminal.
Refer to Figure 2.1 to see a picture of the GNOME Terminal. It was accessed in a
GNOMEdesktopenvironmentonaCentOSdistribution.
In the Unity desktop environment, accessing the GNOME terminal takes a little more
effort. The simplest access method is Dash⇨Search and type Terminal. The GNOME
terminalshowsupintheDashhomeareaasanapplicationnamed Terminal.Clickthat
icontoopentheGNOMEterminalemulator.
Tip
InsomeLinuxdistributiondesktopenvironments,suchasUbuntu’sUnity,youcan
quicklyaccesstheGNOMEterminalusingtheshortcutkeycombinationCtrl+Alt+T.
IntheKDEdesktopenvironment,theKonsoleterminalemulatoristhedefaultemulator.
Therefore,youmustdigdownthroughthemenustoaccessGNOMETerminal.Startwith
the icon labeled Kickoff Application Launcher in the lower-left corner of the screen
andthenclickApplications⇨Utilities⇨Terminal.
In most desktop environments, you can create a launcher for accessing GNOME
Terminal. A launcher is an icon you create on your desktop that allows you to start a
chosen application. This is a great feature that allows you to quickly access a terminal
emulator in the graphical desktop. It is especially helpful if you do not want to use
shortcutkeysortheshortcutkeyfeatureisnotavailableinyourdesktopenvironmentof
choice.
Forexample,intheGNOMEdesktopenvironment,tocreatealauncher,right-clickyour
mouse in the middle of the desktop area; a drop-down menu appears. Select Create
Launcher… from the menu; the Create Launcher application window opens. In the Type
field,selectApplication.TypeanameforyouriconintheNamefield.IntheCommand
field,typegnome-terminal.Click Oktosaveyournewlauncher.Aniconwiththename
yougavethelaunchernowappearsonyourdesktop.Double-clickittoopentheGNOME
terminalemulator.
Note
Whenyoutypegnome-terminalintheCommandfield,youaretypingtheshell
commandforstartingtheGNOMEterminalemulator.YoulearninChapter3howto
addspecialoptionstocommands,suchasgnome-terminal,toprovidespecial
configurationoptions,andhowtoviewalltheoptionsavailabletoyou.
Severalconfigurationoptionsareprovidedbymenusandshort-cutkeysintheapplication,
whichyoucanapplyafteryougettheGNOMEterminalemulationstarted.Understanding
theseoptionscanenhanceyourGNOMETerminalCLIexperience.
TheMenuBar
The GNOME Terminal menu bar contains the configuration and customization options
youneedtomakeyourGNOMETerminaljustthewayyouwantit.Thefollowingtables
briefly describe the different configuration options in the menu bar and shortcut keys
associatedwiththeoptions.
Note
AsyoureadthroughtheseGNOMETerminalmenuoptions,keepinmindthatyour
Linuxdistribution’sGNOMETerminalmayhaveslightlydifferentmenuoptions
available.ThisisbecauseseveralLinuxdistributionsuseolderversionsofGNOME
Terminal.
Table2.4 shows the configuration options available within the GNOME Terminal File
menusystem.The FilemenuitemcontainsitemstocreateandmanageyouroverallCLI
terminalsessions.
Table2.4TheFileMenu
Name
ShortcutKey
Description
StartsanewshellsessioninanewGNOMETerminal
window
Startsanewshellsessioninanewtabintheexisting
GNOMETerminalwindow
Customizesasessionandsavesasaprofile,whichcanbe
recalledforlateruse
Open
Terminal
Shift+Ctrl+N
OpenTab
Shift+Ctrl+T
New
Profile
None
Save
Contents
None
Savesthescrollbackbuffercontentstoatextfile
CloseTab
Shift+Ctrl+W
Closesthecurrenttabsession
Close
Window
Shift+Ctrl+Q
ClosesthecurrentGNOMETerminalsession
Noticethat,asinanetworkbrowser,youcanopennewtabswithintheGNOMETerminal
session to start a whole new CLI session. Each tab session is considered to be an
independentCLIsession.
Tip
YoudonothavetoclickthroughthemenutoreachoptionsintheFilemenu.Mostof
theitemsarealsoavailablebyright-clickinginthesessionarea.
TheEditmenucontainsitems,showninTable2.5,forhandlingtextwithinthetabs.You
canuseyourmousetocopyandpastetextanywherewithinthesessionwindow.
Table2.5TheEditMenu
Name
ShortcutKey
Description
Copy
Shift+Ctrl+C
CopiesselectedtexttotheGNOMEclipboard
Paste
Shift+Ctrl+V PastestextfromtheGNOMEclipboardintoasession
PasteFilenames
Properlypastescopiedfilenamesandtheirpaths
SelectAll
None
Selectsoutputintheentirescrollbackbuffer
Profiles
None
Adds,deletes,ormodifiesGNOMETerminalprofiles
Keyboard
CreateskeycombinationstoquicklyaccessGNOME
None
Shortcuts
Terminalfeatures
Profile
Preferences
None
Editsthecurrentsessionprofile
The Paste Filenames menu option is available only in later versions of GNOME
Terminal.Therefore,youmaynotseethatmenuoptiononyoursystem.
The Viewmenu,showninTable2.6,containsitemsforcontrollinghowtheCLIsession
windowsappear.Theseoptionscanbehelpfulforindividualswithvisualimpairment.
Table2.6TheViewMenu
Name
Shortcut
Key
Description
Show
Menubar
None
Toggleson/offthemenubardisplay
FullScreen
F11
ZoomIn
Ctrl++
Ctrl+Ctrl+0
ZoomOut
NormalSize
Toggleson/offtheterminalwindowfillingtheentire
desktop
Enlargesthefontsizeinthewindowincrementally
Reducesthefontsizeinthewindowincrementally
Returnsthefontsizetodefault
Beawarethatifyoutoggleoffthemenubardisplay,thesession’smenubardisappears.
However, you can easily get the menu bar to display again by right-clicking in any
terminalsessionwindowandtogglingontheShowMenubaroption.
The Search menu, shown in Table 2.7, contains items for conducting simple searches
withintheterminalsession.Thesesearchesaresimilartoonesyoumayhaveconductedin
anetworkbrowserorwordprocessor.
Table2.7TheSearchMenu
Name
ShortcutKey
Description
Find
Shift+Ctrl+F OpensFindwindowtoprovidedesignatedtextsearchoptions
Searchesforwardfromcurrentterminalsessionlocationfor
FindNext Shift+Ctrl+H
designatedtext
Find
Searchesbackwardfromcurrentterminalsessionlocationfor
Shift+Ctrl+G
Previous
designatedtext
The Terminal menu, shown in Table 2.8, contains options for controlling the terminal
emulationsessionfeatures.Therearenoshortcutkeystoaccesstheseitems.
Table2.8TheTerminalMenu
Name
SetTitle
Description
Switchestoanewprofileconfiguration
Modifiessessiontabtitlebarsetting
SetCharacter
Encoding
Selectscharactersetusedtosendanddisplaycharacters
Reset
Sendsresetterminalsessioncontrolcode
Sendsresetterminalsessioncontrolcodeandclearsterminal
sessionscreen
Listswindowsizesforadjustingthecurrentterminalwindowsize
ChangeProfile
ResetandClear
WindowSizeList
The Resetoptionisextremelyuseful.Oneday,youmayaccidentlycauseyourterminal
session to display random characters and symbols. When this occurs, the text is
unreadable. It is typically caused by displaying a non-text file to the screen. You can
quicklygettheterminalsessionbacktonormalbyselectingResetorResetandClear.
TheTabsmenu,showninTable2.9,providesitemsforcontrollingthelocationofthetabs
andselectingwhichtabisactive.Thismenudisplaysonlywhenyouhavemorethanone
tabsessionopen.
Table2.9TheTabsMenu
Name
ShortcutKey
Ctrl+Page
NextTab
Down
Description
Makesthenexttabinthelistactive
Previous
Tab
Ctrl+PageUp
Makestheprevioustabinthelistactive
MoveTab
Left
Shift+Ctrl+Page
Up
Shufflesthecurrenttabinfrontoftheprevioustab
MoveTab
Right
Shift+Ctrl+Page
Down
Detach
Tab
None
TabList
None
Terminal
List
None
Shufflesthecurrenttabinfrontofthenexttab
RemovesthetabandstartsanewGNOMETerminalwindow
usingthistabsession
Liststhecurrentlyrunningtabs(Selectatabtojumptothat
session.)
Liststhecurrentlyrunningterminals(Selectaterminalto
jumptothatsession.Thisisdisplayedonlyifmultiple
windowsessionsareopen.)
Finally, the Help menu contains two menu options. Contents provides a full GNOME
Terminal manual so you can research individual GNOME Terminal items and features.
TheAboutoptionshowsyouthecurrentGNOMETerminalversionthat’srunning.
Besides the GNOME terminal emulator package, another commonly used package is
Konsole Terminal. In many ways, Konsole Terminal is similar to GNOME Terminal.
However,enoughdifferencesexisttowarrantitsownsection.
UsingtheKonsoleTerminalEmulator
The KDE Desktop Project created its own terminal emulation package called Konsole
Terminal.TheKonsolepackageincorporatesbasicterminalemulationfeatures,alongwith
more advanced ones expected from a graphical application. This section describes
KonsoleTerminalfeaturesandshowsyouhowtousethem.
AccessingtheKonsoleTerminal
TheKonsoleTerminalisthedefaultterminalemulatorfortheKDEdesktopenvironment.
You can easily access it via the KDE environment’s menu system. In other desktop
environments,accessingtheKonsoleTerminalcanbealittlemoredifficult.
In the KDE desktop environment, you can access the Konsole Terminal by clicking the
iconlabeledKickoffApplicationLauncherinthelower-leftcornerofthescreen.Then
clickApplications⇨System⇨Terminal(Konsole).
Note
YoumayseetwoterminalmenuoptionswithintheKDEmenuenvironment.Ifyou
do,theTerminalmenuoptionwiththewordsKonsolebeneathitistheKonsole
terminal.
In the GNOME desktop environment, the Konsole terminal is typically not installed by
default.IfKonsoleTerminalhasbeeninstalled,youcanaccessitviatheGNOMEmenu
system. In the upper-left corner of the window, click Applications⇨System
Tools⇨Konsole.
Note
YoumaynothavetheKonsoleterminalemulationpackageinstalledonyoursystem.
Ifyouwouldliketoinstallit,readthroughChapter9tolearnhowtoinstallsoftware
viathecommandline.
In the Unity desktop environment, if Konsole has been installed, you can access it via
Dash⇨SearchandtypeKonsole.TheKonsoleTerminalshowsupintheDashhomearea
asanapplicationnamedKonsole.ClickthaticontoopentheKonsoleterminalemulator.
Figure2.4showstheKonsoleTerminal.ItwasaccessedonaKDEdesktopenvironment
inaCentOSLinuxdistribution.
Figure2.4TheKonsoleTerminal
Remember that, in most desktop environments, you can create a launcher to access
applications such as the Konsole Terminal. The command you need to type for the
launcher to start up the Konsole terminal emulator is konsole. Also, if the Konsole
Terminal is installed, you can start it from another terminal emulator by typing konsole
andpressingEnter.
The Konsole Terminal, similar to GNOME Terminal, has several configuration options
provided by menus and shortcut keys. The following section describes these various
options.
TheMenuBar
TheKonsoleTerminalmenubarcontainstheconfigurationandcustomizationoptionsyou
needtoeasilyviewandchangefeaturesinyourterminalemulationsession.Thefollowing
tablesbrieflydescribethemenuoptionsandassociatedshortcutkeys.
Tip
TheKonsoleTerminalprovidesasimplemenuwhenyouright-clickintheactive
sessionarea.Severalmenuitemsareavailableinthiseasy-to-accessmenu.
TheFilemenu,showninTable2.10,providesoptionsforstartinganewtabinthecurrent
windoworinanewwindow.
Table2.10TheFileMenu
Name
ShortcutKey
NewTab
Ctrl+Shift+N
NewWindow
Ctrl+Shift+M
Shell
None
Description
Startsanewshellsessioninanewtabintheexisting
KonsoleTerminalwindow
StartsanewshellsessioninanewKonsoleTerminal
window
Opensthedefaultprofile,Shell
OpenBrowser
Here
None
Opensthedefaultfilebrowserapplication
CloseTab
Ctrl+Shift+W
Ctrl+Shift+Q
Closesthecurrenttabsession
QuitstheKonsoleTerminalemulationapplication
Quit
WhenyoufirststarttheKonsoleTerminal,theonlyprofilelistedinthemenuisShell.As
moreprofilesarecreatedandsaved,theirnamesappearinthemenulist.
Note
AsyoureadthroughtheseKonsoleTerminalmenuoptions,keepinmindthatyour
Linuxdistribution’sKonsoleTerminalmayhaveverydifferentmenuoptions
available.ThisisbecausesomeLinuxdistributionshavekeptolderversionsofthe
KonsoleTerminalemulationpackage.
The Edit menu, shown in Table2.11, provides options for handling text in the session.
Also,managingtabnamesisinthisoptionslist.
Table2.11TheEditMenu
Name
ShortcutKey
Copy
Ctrl+Shift+C
Paste
Ctrl+Shift+V
RenameTab Ctrl+Alt+S
Description
CopiesselectedtexttotheKonsoleclipboard
PastestextfromtheKonsoleclipboardintoasession
Modifiessessiontabtitlebarsetting
CopyInput
To
None
Starts/stopssessioninputcopiestochosenadditionalsessions
Clear
Display
None
Clearstheterminalsessionscreen
Clear&
Reset
None
Clearstheterminalsessionscreenandsendstheresetterminal
sessioncontrolcode
Konsoleprovidesanexcellentmethodfortrackingwhatfunctionistakingplaceineach
tabsession.UsingtheRenameTabmenuoption,youcannameatabtomatchitscurrent
task.Thishelpsintrackingwhichopentabsessionisperformingwhatfunction.
The View menu, shown in Table 2.12, contains items for controlling individual session
views in the Konsole Terminal window. In addition, options are available that aid in
monitoringterminalsessionactivity.
Table2.12TheViewMenu
Name
ShortcutKey
SplitView
None
DetachView
Ctrl+Shift+H
ShowMenuBar
None
FullScreen
Mode
Ctrl+Shift+F11
Monitorfor
Silence
Ctrl+Shift+I
Description
Controlsthemultipletabsessiondisplaywithinthe
currentKonsoleTerminalwindow
RemovesatabsessionandstartsanewKonsole
Terminalwindowusingthistabsession
Toggleson/offMenubardisplay
Toggleson/offtheterminalwindowfillingtheentire
monitordisplayarea
Toggleson/offaspecialmessagefortabsilence
Monitorfor
Activity
Ctrl+Shift+A
Toggleson/offaspecialmessagefortabactivity
Character
Encoding
None
Selectsthecharactersetusedtosendanddisplay
characters
IncreaseText
Size
Ctrl++
Enlargesthefontsizeinthewindowincrementally
DecreaseText
Size
Ctrl+-
Reducesthefontsizeinthewindowincrementally
The Monitor for Silence menu option is used for indicating tab silence. Tab silence
occurs when no new text appears in the current tab session for 10 seconds. This allows
youtoswitchtoanothertabwhilewaitingforapplicationoutputtostop.
Tabactivity,toggledbytheMonitorforActivityoption,issuesaspecialmessagewhen
new text appears in the tab session. This option allows you to be notified when output
fromanapplicationoccurs.
Konsole retains a history, formally called a scrollbackbuffer, for each tab. The history
containsoutputtextthathasscrolledoutoftheterminalviewingarea.Bydefault,thelast
1,000linesinthescrollbackbufferareretained.The Scrollbackmenu,showninTable
2.13,containsoptionsforviewingthisbuffer.
Table2.13TheScrollbackMenu
Name
ShortcutKey
Description
OpenstheFindwindowatthebottomoftheKonsole
SearchOutput Ctrl+Shift+F
Terminalwindowtoprovidescrollbacktextsearch
options
Findsthenexttextmatchinmorerecentscrollbackbuffer
FindNext
F3
history
Findsthenexttextmatchinolderscrollbackbuffer
FindPrevious
Shift+F3
history
SaveOutput
None
SavesscrollbackbuffercontentstoatextorHTMLfile
Scrollback
OpenstheScrollbackOptionswindowtoconfigure
None
Options
scrollbackbufferoptions
Clear
Scrollback
None
Removesscrollbackbuffercontents
Clear
Scrollback&
Reset
Ctrl+Shift+X
Removesscrollbackbuffercontentsandresetsthe
terminalwindow
You can scroll back through the scrollback buffer by simply using the scrollbar in the
viewing area. Also, you can scroll back line by line by pressing the Shift+Up Arrow or
scrollbackapage(24lines)atatimebypressingShift+PageUp.
The Bookmarksmenuoptions,showninTable2.14,provideawaytomanagebookmarks
set in the Konsole Terminal window. A bookmark enables you to save your active
session’sdirectorylocationandtheneasilyreturnthereineitherthesamesessionoranew
session.
Table2.14TheBookmarksMenu
Name
Shortcut
Key
AddBookmark
Ctrl+Shift+B
BookmarkTabsas
Folder
None
NewBookmarkFolder
None
None
EditBookmarks
Description
Createsanewbookmarkatthecurrentdirectory
location
Createsanewbookmarkforallcurrentterminal
tabsessions
Createsanewbookmarkstoragefolder
Editsexistingbookmarks
The Settings menu, shown in Table 2.15, allows you to customize and manage your
profiles.Also,youcanaddalittlemorefunctionalitytoyourcurrenttabsession.Thereare
noshortcutkeystoaccesstheseitems.
Table2.15TheSettingsMenu
Name
Description
ChangeProfile
Appliestothecurrenttabaselectedprofile
OpenstheEditProfilewindowtoprovideprofileconfiguration
EditCurrentProfile
options
OpenstheManageProfilewindowtoprovideprofile
ManageProfiles
managementoptions
ConfigureShortcuts
CreatesKonsoleTerminalcommandkeyboardshortcuts
Configure
Notifications
CreatescustomKonsoleTerminalschemasandsessions
ConfigureNotificationsallowsyoutoassociatespecificeventsthatcanoccurwithina
session with different actions. When one of the events occurs, the defined action (or
actions)istaken.
The Help menu, shown in Table 2.16, provides the full Konsole handbook (if KDE
handbooks were installed in your Linux distribution) and the standard About Konsole
dialogbox.
Table2.16TheHelpMenu
Name
KonsoleHandbook
What’sThis?
ReportBug
Shortcut
Key
None
Shift+F1
None
Description
ContainsthefullKonsoleHandbook
Containshelpmessagesforterminalwidgets
OpenstheSubmitBugReportform
SwitchApplication
Language
None
OpenstheSwitchApplication’sLanguageform
AboutKonsole
None
DisplaysthecurrentKonsoleTerminalversion
DisplaysthecurrentKDEdesktopenvironment
version
AboutKDE
Rather extensive documentation is provided to help you use the Konsole terminal
emulatorpackage.Inadditiontohelpitems,youareprovidedwithaBugReportformto
submittotheKonsoleTerminaldeveloperswhenyouencounteraprogrambug.
TheKonsoleterminalemulatorpackageisyoungcomparedtoanotherpopularpackage,
xterm.Inthenextsection,weexplorethe“old-timer”xterm.
UsingthextermTerminalEmulator
The oldest and most basic of terminal emulation packages is xterm. The xterm package
hasbeenaroundsincebeforetheoriginaldaysofXWindow,apopulardisplayserver,and
it’softenincludedbydefaultindistributions.
Although xterm is a full terminal emulation package, it doesn’t require many resources
(suchasmemory)tooperate.Becauseofthis,thextermpackageisstillpopularinLinux
distributionsdesignedtorunonolderhardware.Somegraphicaldesktopenvironmentsuse
itasthedefaultterminalemulationpackage.
Although it doesn’t offer many fancy features, the xterm package does one thing
extremely well: It emulates older terminals, such as the Digital Equipment Corporation
(DEC) VT102, VT220, and Tektronix 4014 terminals. For the VT102 and VT220
terminals,xtermcanevenemulatetheVTseriesofcolorcontrolcodes,allowingyouto
usecolorinyourscripts.
Note
TheDECVT102andVT220weredumbtextterminalspopularforconnectingto
Unixsystemsinthe1980sandearly1990s.AVT102/VT220coulddisplaytextand
displayrudimentarygraphicsusingblockmodegraphics.Thisstyleofterminal
accessisstillusedinmanybusinessenvironmentstoday,thuskeepingVT102/VT220
emulationpopular.
Figure2.5 shows what the basic xterm display looks like running on a graphical Linux
desktop.Youcanseeitisverybasic.
Figure2.5ThextermTerminal
Thextermterminalemulatorcanbetrickytofindthesedays.Often,itisnotincludedina
desktopenvironmentgraphicalmenuarrangement.
Accessingxterm
In Ubuntu’s Unity desktop, xterm is installed by default. You can access it via
Dash⇨Searchandtypexterm.xtermshowsupintheDashhomeareaasanapplication
namedXTerm.Clickthaticontoopenthextermterminalemulator.
Note
YoumayseeanotherterminalcalledUXTermwhenyousearchforxtermonUbuntu.
ThisissimplythextermemulatorpackagewithUnicodesupport.
In the GNOME and KDE desktop environment, xterm is not installed by default. You
must install it first (see Chapter 9 for help on installing software packages). After it’s
installed,youmuststartxtermfromanotherterminalemulator.Openaterminalemulator
forCLIaccess,typexterm,andpressEnter.Also,rememberthatyoucancreateyourown
desktoplaunchertostartupxterm.
Thextermpackageallowsyoutosetindividualfeaturesusingcommandlineparameters.
Thefollowingsectionsdiscussthesefeaturesandhowtochangethem.
CommandLineParameters
Thelistofxtermcommandlineparametersisextensive.Youcancontrollotsoffeaturesto
customize the terminal emulation features, such as enabling or disabling individual VT
emulations.
Note
xtermhasahugenumberofconfigurationoptions—somanythattheycannotallbe
coveredhere.Extensivedocumentationisavailableviathebashmanual.Accessing
thebashmanualiscoveredinChapter3.Inaddition,thextermdevelopmentteam
providessomeexcellenthelponitswebsite:http://invisibleisland.net/xterm/.
You can invoke certain configuration options by adding a parameter to the xterm
command. For example, to have the xterm emulate a DEC VT100 terminal, type the
commandxterm -ti vt100 and press Enter. Table2.17 shows some parameters you can
includewheninvokingthextermterminalemulatorsoftware.
Table2.17xtermCommandLineParameters
Parameter
Description
-bgcolor
Specifiesthecolortousefortheterminalbackground
-fbfont
Specifiesthefonttouseforboldtext
-fgcolor
Specifiesthecolortousefortheforegroundtext
-fnfont
Specifiesthefonttousefortext
-fwfont
Specifiesthefonttouseforwidetext
-lffilename
Specifiesthefilenametouseforscreenlogging
-mscolor
Specifiesthecolorusedforthetextcursor
-namename Specifiesthenameoftheapplicationthatappearsinthetitlebar
-titerminal
Specifiestheterminaltypetoemulate
Somextermcommandlineparametersuseaplussign(+)orminussign(-)tosignifyhow
a feature is set. A plus sign may turn a feature on, while a minus sign turns it off.
However, the opposite can be true as well. A plus sign may disable a feature, while a
minussignenablesit,suchaswhenusingthe bcparameter.Table2.18listssomeofthe
morecommonfeaturesyoucansetusingthe+/-commandlineparameters.
Table2.18xterm+/-CommandLineParameters
Parameter
Description
ah
Enables/disableshighlightedtextcursor
aw
Enables/disablesauto-line-wrap
bc
Enables/disablestextcursorblinking
cm
Enables/disablesrecognitionofANSIcolorchangecontrolcodes
fullscreen
Enables/disablesfullscreenmode
j
Enables/disablesjumpscrolling
l
mb
rv
t
Enables/disablesloggingscreendatatoalogfile
Enables/disablesmarginbell
Enables/disablesreversevideocolors
Enables/disablesTektronixmode
It is important to note that not all implementations of xterm support all these command
lineparameters.Youcandeterminewhichparametersyourxtermimplementsbyusingthe
-helpparameterwhenyoustartxtermonyoursystem.
Nowthatyouhavebeenintroducedtothreeterminalemulatorpackages,thebigquestion
iswhichisthebestterminalemulatortouse?Thereisnodefiniteanswertothatquestion.
Which terminal emulator package you use depends upon your individual needs and
desires.Butitisgreattohavesomanychoices.
Summary
TostartlearningLinuxcommandlinecommands,youneedaccesstoaCLI.Intheworld
of graphical interfaces, this can sometimes be challenging. This chapter discussed
differentinterfacesyoushouldconsidertogettotheLinuxcommandline.
First,thischapterdiscussedthedifferencebetweenaccessingtheCLIviaavirtualconsole
terminal (a terminal outside the GUI) and a graphical terminal emulation package (a
terminalinsidetheGUI).Wetookabrieflookatthebasicdifferencesbetweenthesetwo
accessmethods.
Next, we explored in detail accessing the CLI via a virtual console terminal, including
specifics on how to change console terminal configuration options such as background
color.
Afterlookingatvirtualconsoleterminals,thechaptertraveledthroughaccessingtheCLI
viaagraphicalterminalemulator.Primarily,wecoveredthreedifferenttypesofterminal
emulators:GNOMETerminal,KonsoleTerminal,andxterm.
This chapter also covered the GNOME desktop project’s GNOME terminal emulation
package. GNOME Terminal is typically installed by default on the GNOME desktop
environment.Itprovidesconvenientwaystosetmanyterminalfeaturesviamenuoptions
andshortcutkeys.
We also covered the KDE desktop project’s Konsole terminal emulation package. The
Konsole Terminal is typically installed by default on the KDE desktop environment. It
providesseveralnicefeatures,suchastheabilitytomonitoraterminalforsilence.
Finally, we covered the xterm terminal emulator package. xterm was the first terminal
emulatoravailableforLinux.ItcanemulateolderterminalhardwaresuchastheVTand
Tektronixterminals.
Inthenextchapter,youstartlookingattheLinuxcommandlinecommands.Itwalksyou
throughthecommandsnecessarytonavigatearoundtheLinuxfilesystem,andtocreate,
delete,andmanipulatefiles.
Chapter3
BasicbashShellCommands
InThisChapter
1. Interactingwiththeshell
2. Usingthebashmanual
3. Traversingthefilesystem
4. Listingfilesanddirectories
5. Managingfilesanddirectories
6. Viewingfilecontents
ThedefaultshellusedinmanyLinuxdistributionsistheGNUbashshell.This
chapterdescribesthebasicfeaturesavailableinthebashshell,suchasthebash
manual,tabauto-completionandhowtodisplayafile’scontents.Youwillwalk
throughhowtoworkwithLinuxfilesanddirectoriesusingthebasiccommands
providedbythebashshell.Ifyou’realreadycomfortablewiththebasicsintheLinux
environment,feelfreetoskipthischapterandcontinuewithChapter4toseemore
advancedcommands.
StartingtheShell
TheGNUbashshellisaprogramthatprovidesinteractiveaccesstotheLinuxsystem.It
runsasaregularprogramandisnormallystartedwheneverauserlogsintoaterminal.
TheshellthatthesystemstartsdependsonyouruserIDconfiguration.
The/etc/passwdfilecontainsalistofallthesystemuseraccounts,alongwithsomebasic
configurationinformationabouteachuser.Here’sasampleentryfroma/etc/passwdfile:
christine:x:501:501:ChristineBresnahan:/home/christine:/bin/bash
Eachentryhassevendatafields,withfieldsseparatedbycolons.Thesystemusesthedata
inthesefieldstoassignspecificfeaturesfortheuser.Mostoftheseentriesarediscussedin
moredetailinChapter7.Fornow,justpayattentiontothelastfield,whichspecifiesthe
user’sshellprogram.
Note
ThoughthefocusisontheGNUbashshell,additionalshellsarereviewedinthis
book.Chapter23coversworkingwithalternativeshells,suchasdashandtcsh.
In the earlier /etc/passwd sample entry, the user christine has /bin/bash set as her
defaultshellprogram.Thismeanswhen christinelogsintotheLinuxsystem,thebash
shellprogramisautomaticallystarted.
Although the bash shell program is automatically started at login, whether a shell
command line interface (CLI) is presented depends on which login method is used. If a
virtualconsoleterminalisusedtologin,theCLIpromptisautomaticallypresented,and
youcanbegintotypeshellcommands.However,ifyoulogintotheLinuxsystemviaa
graphicaldesktopenvironment,youneedtostartagraphicalterminalemulatortoaccess
theshellCLIprompt.
UsingtheShellPrompt
AfteryoustartaterminalemulationpackageorlogintoaLinuxvirtualconsole,youget
accesstotheshellCLIprompt.Thepromptisyourgatewaytotheshell.Thisistheplace
whereyouentershellcommands.
Thedefaultpromptsymbolforthebashshellisthedollarsign($).Thissymbolindicates
that the shell is waiting for you to enter text. Different Linux distributions use different
formatsfortheprompt.OnthisUbuntuLinuxsystem,theshellpromptlookslikethis:
christine@server01:∼$
OntheCentOSLinuxsystem,itlookslikethis:
[christine@server01∼]$
Besidesactingasyouraccesspointtotheshell,thepromptcanprovideadditionalhelpful
information. In the two preceding examples, the current user ID name, christine, is
shownintheprompt.Also,thenameofthesystemisshown,server01.Youlearnlaterin
thischapteraboutadditionalitemsshownintheprompt.
Tip
IfyouarenewtotheCLI,keepinmindthat,afteryoutypeinashellcommandatthe
prompt,youneedtopresstheEnterkeyfortheshelltoactuponyourcommand.
The shell prompt is not static. It can be changed to suit your needs. Chapter 6, “Using
LinuxEnvironmentVariables,”coversmodifyingyourshellCLIpromptconfiguration.
ThinkoftheshellCLIpromptasahelpmate,assistingyouwithyourLinuxsystem,giving
you helpful insights, and letting you know when the shell is ready for new commands.
AnotherhelpfulitemintheshellisthebashManual.
InteractingwiththebashManual
Most Linux distributions include an online manual for looking up information on shell
commands,aswellaslotsofotherGNUutilitiesincludedinthedistribution.Youshould
become familiar with the manual, because it’s invaluable for working with commands,
especiallywhenyou’retryingtofigureoutvariouscommandlineparameters.
The man command provides access to the manual pages stored on the Linux system.
Enteringthe mancommandfollowedbyaspecificcommandnameprovidesthatutility’s
manualentry.Figure3.1showsanexampleoflookingupthe xtermcommand’smanual
pages.Thispagewasreachedbytypingthecommandmanxterm.
Figure3.1Manualpagesforthextermcommand
NoticethextermcommandDESCRIPTIONparagraphsinFigure3.1.Theyarerathersparse
and full of technical jargon. The bash manual is not a step-by-step guide, but instead a
quickreference.
Tip
Ifyouarenewtothebashshell,youmayfindthatthemanpagesarenotveryhelpful
atfirst.However,getintothehabitofusingthem,especiallytoreadthefirst
paragraphortwoofacommand’sDESCRIPTIONsection.Eventually,youwilllearn
thetechnicallingo,andthemanpageswillbecomemorehelpfultoyou.
Whenyouusethe mancommandtoviewacommand’smanualpages,theyaredisplayed
with something called a pager. A pager is a utility that allows you to page through
displayedtext.Thus,youcanpagethroughthemanpagesbypressingthespacebar,oryou
cangolinebylineusingtheEnterkey.Inaddition,youcanusethearrowkeystoscroll
forwardandbackwardthroughthemanpagetext(assumingthatyourterminalemulation
packagesupportsthearrowkeyfunctions).
Whenyouarefinishedwiththemanpages,presstheqkeytoquit.Whenyouquittheman
pages, you receive a shell CLI prompt, indicating the shell is waiting for your next
command.
Tip
Thebashmanualevenhasreferenceinformationonitself.Typemanmantosee
manualpagesconcerningthemanpages
The manual page divides information about a command into separate sections. Each
sectionhasaconventionalnamingstandardasshowninTable3.1.
Table3.1TheLinuxmanPageConventionalSectionNames
Section
Description
Name
Displayscommandnameandashortdescription
Synopsis
Showscommandsyntax
Configuration
Providesconfigurationinformation
Description
Describescommandgenerally
Options
Describescommandoption(s)
ExitStatus
Definescommandexitstatusindicator(s)
ReturnValue
Describescommandreturnvalue(s)
Errors
Providescommanderrormessages
Environment
Describesenvironmentvariable(s)used
Files
Definesfilesusedbycommand
Versions
Describescommandversioninformation
ConformingTo
Providesstandardsfollowed
Notes
Describesadditionalhelpfulcommandmaterial
Bugs
Providesthelocationtoreportfoundbugs
Example
Showscommanduseexamples
Authors
Providesinformationoncommanddevelopers
Copyright
Definescommandcodecopyrightstatus
SeeAlso
Referssimilaravailablecommands
Not every command’s man page has all the section names described in Table3.1.Also,
somecommandshavesectionnamesthatarenotlistedintheconventionalstandard.
Tip
Whatifyoucan’trememberthecommandname?Youcansearchthemanpages
usingkeywords.Thesyntaxisman-kkeyword.Forexample,tofindcommands
dealingwiththeterminals,youtypeman-kterminal.
In addition to the conventionally named sections for a man page, there are man page
sectionareas.Eachsectionareahasanassignednumber,startingat1andgoingto9;they
arelistedinTable3.2.
Table3.2TheLinuxmanPageSectionAreas
SectionNumber
AreaContents
1
Executableprogramsorshellcommands
2
Systemcalls
3
Librarycalls
4
Specialfiles
5
Fileformatsandconventions
6
Games
7
Overviews,conventions,andmiscellaneous
8
Superuserandsystemadministrationcommands
9
Kernelroutines
Typically, the man utility provides the lowest numbered content area for the command.
For example, looking back to Figure3.1 where the command man xterm was entered,
noticethatintheupper-leftandupper-rightdisplaycorners,thewordXTERMisfollowedby
a number in parentheses, (1). This means the man pages displayed are coming from
contentarea1(executableprogramsorshellcommands).
Occasionally,acommandhasmanpagesinmultiplesectioncontentareas.Forexample,
thereisacommandcalledhostname.Themanpagescontaininformationonthecommand
aswellasanoverviewsectiononsystemhostnames.Toseethepagesdesired,youtype
mansection#topic.Forthecommand’smanpagesinsection1,typeman1hostname.
Fortheoverviewmanpagesinsection7,typeman7hostname.
Youcanalsostepthroughanintroductiontothevarioussectioncontentareasbytyping
man1introtoreadaboutsection1,man2introtoreadaboutsection2,man3introto
readaboutsection3,andsoon.
Themanpagesarenottheonlyreference.Therearealsotheinformationpagescalledinfo
pages.Youcanlearnabouttheinfopagesbytypinginfoinfo.
Inaddition,mostcommandsacceptthe-helpor—helpoption.Forexample,youcantype
hostname-helptoseeahelpscreen.Formoreinformationonusinghelp,typehelphelp.
(Seeapatternhere?)
Obviously, several helpful resources are available for reference. However, many basic
shell concepts still need detailed explanation. In the next section, we cover navigating
throughtheLinuxfilesystem.
NavigatingtheFilesystem
When you log into the system and reach the shell command prompt, you are usually
placedinyourhomedirectory.Often,youwanttoexploreotherareasintheLinuxsystem
besides just your home directory. This section describes how to do that using shell
commands.Tostart,youneedtotakeatourofjustwhattheLinuxfilesystemlookslikeso
youknowwhereyouaregoing.
LookingattheLinuxfilesystem
If you’re new to the Linux system, you may be confused by how it references files and
directories,especiallyifyou’reusedtothewaytheMicrosoftWindowsoperatingsystem
doesthat.BeforeexploringtheLinuxsystem,ithelpstohaveanunderstandingofhowit’s
laidout.
Thefirstdifferenceyou’llnoticeisthatLinuxdoesnotusedrivelettersinpathnames.In
theWindowsworld,thephysicaldrivesinstalledonthecomputerdeterminethepathname
ofthefile.Windowsassignsalettertoeachphysicaldiskdrive,andeachdrivecontainsits
owndirectorystructureforaccessingfilesstoredonit.
Forexample,inWindowsyoumaybeusedtoseeingthefilepathssuchas:
c:\Users\Rich\Documents\test.doc
The Windows file path tells you exactly which physical disk partition contains the file
namedtest.doc.Forexample,ifyousavedtest.doconaflashdrive,designatedbytheJ
drive,thefilepathwouldbeJ:∖test.doc.Thispathindicatesthatthefileislocatedatthe
rootofthedriveassignedtheletterJ.
ThisisnotthemethodusedbyLinux.Linuxstoresfileswithinasingledirectorystructure,
called a virtual directory. The virtual directory contains file paths from all the storage
devicesinstalledonthecomputer,mergedintoasingledirectorystructure.
The Linux virtual directory structure contains a single base directory, called the root.
Directoriesandfilesbeneaththerootdirectoryarelistedbasedonthedirectorypathused
togettothem,similartothewayWindowsdoesit.
Tip
You’llnoticethatLinuxusesaforwardslash(/)insteadofabackwardslash(∖)to
denotedirectoriesinfilepaths.ThebackslashcharacterinLinuxdenotesanescape
characterandcausesallsortsofproblemswhenyouuseitinafilepath.Thismay
takesomegettingusedtoifyou’recomingfromaWindowsenvironment.
InLinux,youwillseefilepathssimilartothefollowing:
/home/Rich/Documents/test.doc
Thisindicatesthefile test.docisinthedirectory Documents,underthedirectory rich,
which is contained in the directory home. Notice that the path doesn’t provide any
informationastowhichphysicaldiskthefileisstoredon.
The tricky part about the Linux virtual directory is how it incorporates each storage
device.ThefirstharddriveinstalledinaLinuxsystemiscalledtherootdrive.Theroot
drivecontainsthevirtualdirectorycore.Everythingelsebuildsfromthere.
On the root drive, Linux can use special directories as mountpoints. Mount points are
directoriesinthevirtualdirectorywhereyoucanassignadditionalstoragedevices.Linux
causes files and directories to appear within these mount point directories, even though
theyarephysicallystoredonadifferentdrive.
Oftensystemfilesarephysicallystoredontherootdrive.Userfilesaretypicallystoredon
aseparatedriveordrives,asshowninFigure3.2.
Figure3.2TheLinuxfilestructure
Figure3.2showstwoharddrivesonthecomputer.Oneharddriveisassociatedwiththe
rootofthevirtualdirectory(indicatedbyasingleforwardslash).Otherharddrivescanbe
mountedanywhereinthevirtualdirectorystructure.Inthisexample,thesecondharddrive
ismountedatthelocation/home,whichiswheretheuserdirectoriesarelocated.
TheLinuxfilesystemstructureoriginallyevolvedfromtheUnixfilestructure.InaLinux
filesystem,commondirectorynamesareusedforcommonfunctions.Table3.3listssome
ofthemorecommonLinuxvirtualtop-leveldirectorynamesandtheircontents.
Table3.3CommonLinuxDirectoryNames
Directory
Usage
/
rootofthevirtualdirectory,wherenormally,nofilesareplaced
/bin
binarydirectory,wheremanyGNUuser-levelutilitiesarestored
/boot
bootdirectory,wherebootfilesarestored
/dev
devicedirectory,whereLinuxcreatesdevicenodes
/etc
systemconfigurationfilesdirectory
/home
homedirectory,whereLinuxcreatesuserdirectories
/lib
librarydirectory,wheresystemandapplicationlibraryfilesarestored
/media mediadirectory,acommonplaceformountpointsusedforremovablemedia
mountdirectory,anothercommonplaceformountpointsusedforremovable
/mnt
media
optionaldirectory,oftenusedtostorethird-partysoftwarepackagesanddata
/opt
files
/proc
processdirectory,wherecurrenthardwareandprocessinformationisstored
/root
roothomedirectory
/sbin
systembinarydirectory,wheremanyGNUadmin-levelutilitiesarestored
/run
rundirectory,whereruntimedataisheldduringsystemoperation
/srv
servicedirectory,wherelocalservicesstoretheirfiles
/sys
systemdirectory,wheresystemhardwareinformationfilesarestored
/tmp
temporarydirectory,wheretemporaryworkfilescanbecreatedanddestroyed
userbinarydirectory,wherethebulkofGNUuser-levelutilitiesanddatafiles
/usr
arestored
/var
variabledirectory,forfilesthatchangefrequently,suchaslogfiles
The common Linux directory names are based upon the Filesystem Hierarchy Standard
(FHS). Many Linux distributions maintain compliance with FHS. Therefore, you should
beabletoeasilyfindfilesonanyFHS-compliantLinuxsystems.
Note
TheFHSisoccasionallyupdated.YoumayfindthatsomeLinuxdistributionsare
stillusinganolderFHSstandard,whileotherdistributionsonlypartiallyimplement
thecurrentstandard.TokeepuptodateontheFHSstandard,visititsofficialhomeat
http://www.pathname.com/fhs/.
WhenyoulogintoyoursystemandreachashellCLIprompt,yoursessionstartsinyour
homedirectory.Yourhomedirectoryisauniquedirectoryassignedtoyouruseraccount.
When a user account is created, the system normally assigns a unique directory for the
account(seeChapter7).
Youcanmovearoundthevirtualdirectoryusingagraphicalinterface.However,tomove
aroundthevirtualdirectoryfromaCLIprompt,youneedtolearntousethecdcommand.
Traversingdirectories
You use the change directory command (cd) to move your shell session to another
directory in the Linux filesystem. The cd command syntax is pretty simplistic: cd
destination.
Thecdcommandmaytakeasingleparameter,destination,whichspecifiesthedirectory
nameyouwanttogoto.Ifyoudon’tspecifyadestinationonthe cdcommand,ittakes
youtoyourhomedirectory.
Thedestinationparametercanbeexpressedusingtwodifferentmethods.Onemethodis
using an absolute directory reference. The other method uses a relative directory
reference.
Thefollowingsectionsdescribeeachofthesemethods.Thedifferencesbetweenthesetwo
methodsareimportanttounderstandasyoutraversethefilesystem.
Usingabsolutedirectoryreferences
Youcanreferenceadirectorynamewithinthevirtualdirectorysystemusinganabsolute
directoryreference.Theabsolutedirectoryreferencedefinesexactlywherethedirectory
is in the virtual directory structure, starting at the root. Think of the absolute directory
referenceasthefullnameforadirectory.
An absolute directory reference always begins with a forward slash (/), indicating the
virtualdirectorysystem’sroot.Thus,toreferenceuserbinaries,containedwithinthe bin
directorystoredwithintheusrdirectory,youwoulduseanabsolutedirectoryreferenceas
follows:
/usr/bin
Withtheabsolutedirectoryreference,there’snodoubtastoexactlywhereyouwanttogo.
Tomovetoaspecificlocationinthefilesystemusingtheabsolutedirectoryreference,you
justspecifythefullpathnameinthecdcommand:
christine@server01:∼$cd/usr/bin
christine@server01:/usr/bin$
Noticeintheprecedingexamplethatthepromptoriginallyhadatilde(∼)init.Afterthe
changetoanewdirectoryoccurred,thetildewasreplacedby /usr/bin.Thisiswherea
CLI prompt can help you keep track of where you are in the virtual directory structure.
The tilde indicates that your shell session is located in your home directory. After you
moveoutofyourhomedirectory,theabsolutedirectoryreferenceisshownintheprompt,
iftheprompthasbeenconfiguredtodoso.
Note
IfyourshellCLIpromptdoesnotshowyourshellsession’scurrentlocation,thenit
hasnotbeenconfiguredtodoso.Chapter6showsyouhowtomakeconfiguration
changes,ifyoudesiremodificationstoyourCLIprompt.
If your prompt has not been configured to show the shell session’s current absolute
directory location, then you can display the location via a shell command. The pwd
command displays the shell session’s current directory location, which is called the
presentworkingdirectory.Anexampleofusingthepwdcommandisshownhere.
christine@server01:/usr/bin$pwd
/usr/bin
christine@server01:/usr/bin$
Tip
Itisagoodhabittousethepwdcommandwheneveryouchangetoanewpresent
workingdirectory.Becausemanyshellcommandsoperateonthepresentworking
directory,youalwayswanttomakesureyouareinthecorrectdirectorybefore
issuingacommand.
You can move to any level within the entire Linux virtual directory structure from any
levelusingtheabsolutedirectoryreference:
christine@server01:/usr/bin$cd/var/log
christine@server01:/var/log$
christine@server01:/var/log$pwd
/var/log
christine@server01:/var/log$
YoucanalsoquicklyjumptoyourhomedirectoryfromanylevelwithintheLinuxvirtual
directorystructure:
christine@server01:/var/log$cd
christine@server01:~$
christine@server01:∼$pwd
/home/christine
christine@server01:∼$
However, if you’re just working within your own home directory structure, often using
absolute directory references can get tedious. For example, if you’re already in the
directory /home/christine, it seems somewhat cumbersome to have to type the
command:
cd/home/christine/Documents
justtogettoyourDocumentsdirectory.Fortunately,there’sasimplersolution.
Usingrelativedirectoryreferences
Relative directory references allow you to specify a destination directory reference
relativetoyourcurrentlocation.Arelativedirectoryreferencedoesn’tstartwithaforward
slash(/).
Instead, a relative directory reference starts with either a directory name (if you’re
traversingtoadirectoryunderyourcurrentdirectory)oraspecialcharacter.Forexample,
ifyouareinyour homedirectoryandwanttomovetoyour Documentssubdirectory,you
canusethecdcommandalongwitharelativedirectoryreference:
christine@server01:∼$pwd
/home/christine
christine@server01:~$
christine@server01:∼$cdDocuments
christine@server01:∼/Documents$pwd
/home/christine/Documents
christine@server01:~/Documents$
In the preceding example, note that no forward slash (/) was used. Instead a relative
directory reference was used and the present work directory was changed from
/home/christineto/home/christine/Documents,withmuchlesstyping.
Alsonoticeintheexamplethatifthepromptisconfiguredtodisplaythepresentworking
directory,itkeepsthetildeinthedisplay.Thisshowsthatthepresentworkingdirectoryis
inadirectoryundertheuser’shomedirectory.
Tip
IfyouarenewtothecommandlineandtheLinuxdirectorystructure,itis
recommendedthatyoustickwithabsolutedirectoryreferencesforawhile.Afteryou
becomemorefamiliarwiththedirectorylayout,switchtousingrelativedirectory
references.
You can use a relative directory reference with the cd command in any directory
containing subdirectories. You can also use a special character to indicate a relative
directorylocation.
Thetwospecialcharactersusedforrelativedirectoryreferencesare:
Thesingledot(.)torepresentthecurrentdirectory
Thedoubledot(..)torepresenttheparentdirectory
Youcanusethesingledot,butitdoesn’tmakesensetouseitwiththecdcommand.Later
inthechapter,youwillseehowanothercommandusesthesingledotforrelativedirectory
referenceseffectively.
Thedoubledotcharacterisextremelyhandywhentryingtotraverseadirectoryhierarchy.
Forexample,ifyouareintheDocumentsdirectoryunderyourhomedirectoryandneedto
gotoyourDownloadsdirectory,alsounderyourhomedirectory,youcandothis:
christine@server01:∼/Documents$pwd
/home/christine/Documents
christine@server01:∼/Documents$cd../Downloads
christine@server01:∼/Downloads$pwd
/home/christine/Downloads
christine@server01:∼/Downloads$
The double dot character takes you back up one level to your home directory; then the
/Downloadsportionofthecommandtakesyoubackdownintothe Downloadsdirectory.
Youcanuseasmanydoubledotcharactersasnecessarytomovearound.Forexample,if
youareinyourhomedirectory(/home/christine)andwanttogotothe /etcdirectory,
youcouldtypethefollowing:
christine@server01:∼$cd../../etc
christine@server01:/etc$pwd
/etc
christine@server01:/etc$
Ofcourse,inacaselikethis,youactuallyhavetodomoretypingratherthanjusttyping
the absolute directory reference, /etc. Thus, use a relative directory reference only if it
makessensetodoso.
Note
It’shelpfultohavealonginformativeshellCLIprompt,asusedinthischapter
section.However,forclaritypurposes,asimple$promptisusedintherestofthe
book’sexamples.
Now that you know how to traverse the directory system and confirm your present
workingdirectory,youcanstarttoexplorewhat’scontainedwithinthevariousdirectories.
The next section takes you through the process of looking at files within the directory
structure.
ListingFilesandDirectories
To see what files are available on the system, use the list command (ls). This section
describesthelscommandandoptionsavailabletoformattheinformationitcandisplay.
Displayingabasiclisting
The lscommandatitsmostbasicformdisplaysthefilesanddirectorieslocatedinyour
currentdirectory:
$ls
DesktopDownloadsMusicPicturesTemplatesVideos
Documentsexamples.desktopmy_scriptPublictest_file
$
Noticethatthe lscommandproducesthelistinginalphabeticalorder(incolumnsrather
thanrows).Ifyou’reusingaterminalemulatorthatsupportscolor,the lscommandmay
also show different types of entries in different colors. The LS_COLORS environment
variablecontrolsthisfeature.(EnvironmentvariablesarecoveredinChapter6).Different
Linux distributions set this environment variable depending on the capabilities of the
terminalemulator.
If you don’t have a color terminal emulator, you can use the -F parameter with the ls
commandtoeasilydistinguishfilesfromdirectories.Usingthe-Fparameterproducesthe
followingoutput:
$ls-F
Desktop/Downloads/Music/Pictures/Templates/Videos/
Documents/examples.desktopmy_script*Public/test_file
$
The-Fparameterflagsthedirectorieswithaforwardslash(/),tohelpidentifytheminthe
listing.Similarly,itflagsexecutablefiles(likethe my_scriptfileintheprecedingcode)
withanasterisk(*),tohelpyoumoreeasilyfindfilesthatcanberunonthesystem.
The basic ls command can be somewhat misleading. It shows the files and directories
containedinthecurrentdirectory,butnotnecessarilyallofthem.Linuxoftenuseshidden
files to store configuration information. In Linux, hidden files are files with filenames
startingwithaperiod(.).Thesefilesdon’tappearinthedefault lslisting.Thus,theyare
calledhiddenfiles.
Todisplayhiddenfilesalongwithnormalfilesanddirectories,usethe-aparameter.Here
isanexampleofusingthe-aparameterwiththelscommand.
$ls-a
..compizexamples.desktopMusictest_file
...config.gconfmy_scriptVideos
.bash_historyDesktop.gstreamer-0.10Pictures.Xauthority
.bash_logout.dmrc.ICEauthority.profile.xsession-errors
.bashrcDocuments.localPublic.xsession-errors.old
.cacheDownloads.mozillaTemplates
$
Allthefilesbeginningwithaperiod,hiddenfiles,arenowshown.Noticethatthreefiles
begin with .bash. These are hidden files that are used by the bash shell environment.
ThesefeaturesarecoveredindetailinChapter6.
The -Rparameterisanotheroptionthe lscommandcanuse.Calledtherecursiveoption,
itshowsfilesthatarecontainedwithinsubdirectoriesinthecurrentdirectory.Ifyouhave
lotsofsubdirectories,thiscanbequitealonglisting.Here’sasimpleexampleofwhatthe
-Rparameterproduces.The-Foptionwastackedontohelpyouseethefiletypes:
$ls-F-R
.:
Desktop/Downloads/Music/Pictures/Templates/Videos/
Documents/examples.desktopmy_script*Public/test_file
./Desktop:
./Documents:
./Downloads:
./Music:
ILoveLinux.mp3*
./Pictures:
./Public:
./Templates:
./Videos:
$
Notice that the -R parameter shows the contents of the current directory, which are the
files from a user’s home directory shown in earlier examples. It also shows each
subdirectory in the user’s home directory and their contents. The only subdirectory
containing a file is the Music subdirectory, and it contains the executable file,
ILoveLinux.mp3.
Tip
Optionparametersdon’thavetobeenteredseparatelyasshowninthenearby
example:ls-F-R.Theycanoftenbecombinedasfollows:ls-FR.
Inthepreviousexample,therewerenosubdirectorieswithinsubdirectories.Iftherehad
been further subdirectories, the -R parameter would have continued to traverse those as
well.Asyoucansee,forlargedirectorystructures,thiscanbecomequitealargeoutput
listing.
Displayingalonglisting
Inthebasiclistings,the lscommanddoesn’tproducemuchinformationabouteachfile.
For listing additional information, another popular parameter is -l. The -l parameter
produces a long listing format, providing more information about each file in the
directory:
$ls-l
total48
drwxr-xr-x2christinechristine4096Apr2220:37Desktop
drwxr-xr-x2christinechristine4096Apr2220:37Documents
drwxr-xr-x2christinechristine4096Apr2220:37Downloads
-rw-r—r—1christinechristine8980Apr2213:36examples.desktop
-rw-rw-r—1christinechristine0May2113:44fall
-rw-rw-r—1christinechristine0May2113:44fell
-rw-rw-r—1christinechristine0May2113:44fill
-rw-rw-r—1christinechristine0May2113:44full
drwxr-xr-x2christinechristine4096May2111:39Music
-rw-rw-r—1christinechristine0May2113:25my_file
-rw-rw-r—1christinechristine0May2113:25my_scrapt
-rwxrw-r—1christinechristine54May2111:26my_script
-rw-rw-r—1christinechristine0May2113:42new_file
drwxr-xr-x2christinechristine4096Apr2220:37Pictures
drwxr-xr-x2christinechristine4096Apr2220:37Public
drwxr-xr-x2christinechristine4096Apr2220:37Templates
-rw-rw-r—1christinechristine0May2111:28test_file
drwxr-xr-x2christinechristine4096Apr2220:37Videos
$
Thelonglistingformatlistseachfileandsubdirectoryonasingleline.Inadditiontothe
filename,thelistingshowsadditionalusefulinformation.Thefirstlineintheoutputshows
thetotalnumberofblockscontainedwithinthedirectory.Afterthat,eachlinecontainsthe
followinginformationabouteachfile(ordirectory):
Thefiletype—suchasdirectory(d),file(-),linkedfile(l),characterdevice(c),or
blockdevice(b)
Thefilepermissions(seeChapter6)
Thenumberoffilehardlinks(Seethesection“LinkingFiles”inChapter7.)
Thefileownerusername
Thefileprimarygroupname
Thefilebytesize
Thelasttimethefilewasmodified
Thefilenameordirectoryname
The-lparameterisapowerfultooltohave.Armedwiththisparameter,youcanseemost
oftheinformationyouneedforanyfileordirectory.
The ls command has lots of parameters that can come in handy as you do file
management.Ifyoutypeattheshellprompt manls,youseeseveralpagesofavailable
parametersforyoutousetomodifythelscommandoutput.
Don’t forget that you can also combine many of the parameters. You can often find a
parameter combination that not only displays the desired output, but also is easy to
remember,suchasls-alF.
Filteringlistingoutput
As you’ve seen in the examples, by default the ls command lists all the non-hidden
directory files. Sometimes, this can be overkill, especially when you’re just looking for
informationonafewfiles.
Fortunately, the ls command also provides a way for you to define a filter on the
commandline.Itusesthefiltertodeterminewhichfilesordirectoriesitshoulddisplayin
theoutput.
Thefilterworksasasimpletext-matchingstring.Includethefilterafteranycommandline
parametersyouwanttouse:
$ls-lmy_script
-rwxrw-r—1christinechristine54May2111:26my_script
$
Whenyouspecifythenameofaspecificfileasthefilter,the ls commandonlyshows
thatfile’sinformation.Sometimes,youmightnotknowtheexactfilenameyou’relooking
for.Thelscommandalsorecognizesstandardwildcardcharactersandusesthemtomatch
patternswithinthefilter:
Aquestionmark(?)torepresentonecharacter
Anasterisk(*)torepresentanynumberofcharacters
The question mark can be used to replace exactly one character anywhere in the filter
string.Forexample:
$ls-lmy_scr?pt
-rw-rw-r—1christinechristine0May2113:25my_scrapt
-rwxrw-r—1christinechristine54May2111:26my_script
$
Thefiltermy_scr?ptmatchedtwofilesinthedirectory.Similarly,theasteriskcanbeused
tomatchzeroormorecharacters:
$ls-lmy*
-rw-rw-r—1christinechristine0May2113:25my_file
-rw-rw-r—1christinechristine0May2113:25my_scrapt
-rwxrw-r—1christinechristine54May2111:26my_script
$
Using the asterisk finds three different files, starting with the name my. As with the
questionmark,youcanplacetheasterisksanywhereinthefilter:
$ls-lmy_s*t
-rw-rw-r—1christinechristine0May2113:25my_scrapt
-rwxrw-r—1christinechristine54May2111:26my_script
$
Usingtheasteriskandquestionmarkinthefilteriscalledfileglobbing.Fileglobbingis
the processing of pattern matching using wildcards. The wildcards are officially called
metacharacter wildcards. You can use more metacharacter wildcards for file globbing
thanjusttheasteriskandquestionmark.Youcanalsousebrackets:
$ls-lmy_scr[ai]pt
-rw-rw-r—1christinechristine0May2113:25my_scrapt
-rwxrw-r—1christinechristine54May2111:26my_script
$
In this example, we used the brackets along with two potential choices for a single
character in that position, aor i. The brackets represent a single character position and
giveyoumultipleoptionsforfileglobbing.Youcanlistchoicesofcharacters,asshownin
theprecedingexample,andyoucanspecifyarangeofcharacters,suchasanalphabetic
range[a-i]:
$ls-lf[a-i]ll
-rw-rw-r—1christinechristine0May2113:44fall
-rw-rw-r—1christinechristine0May2113:44fell
-rw-rw-r—1christinechristine0May2113:44fill
$
Also, you can specify what should not be included in the pattern match by using the
exclamationpoint(!):
$ls-lf[!a]ll
-rw-rw-r—1christinechristine0May2113:44fell
-rw-rw-r—1christinechristine0May2113:44fill
-rw-rw-r—1christinechristine0May2113:44full
$
Fileglobbingisapowerfulfeaturewhensearchingforfiles.Itcanalsobeusedwithother
shellcommandsbesidesls.Youfindoutmoreaboutthislaterinthechapter.
HandlingFiles
The shell provides many file manipulation commands on the Linux filesystem. This
sectionwalksyouthroughthebasicshellcommandsyouneedtohandlefiles.
Creatingfiles
Everyonceinawhileyourunintoasituationwhereyouneedtocreateanemptyfile.For
example,sometimesapplicationsexpectalogfiletobepresentbeforetheycanwritetoit.
Inthesesituations,youcanusethetouchcommandtoeasilycreateanemptyfile:
$touchtest_one
$ls-ltest_one
-rw-rw-r—1christinechristine0May2114:17test_one
$
The touch command creates the new file you specify and assigns your username as the
file owner. Notice in the preceding example that the file size is zero because the touch
commandjustcreatedanemptyfile.
The touch command can also be used to change the modification time. This is done
withoutchangingthefilecontents:
$ls-ltest_one
-rw-rw-r—1christinechristine0May2114:17test_one
$touchtest_one
$ls-ltest_one
-rw-rw-r—1christinechristine0May2114:35test_one
$
Themodificationtimeoftest_oneisnowupdatedto14:35fromtheoriginaltime,14:17.
Tochangeonlytheaccesstime,usethe-aparameterwiththetouchcommand:
$ls-ltest_one
-rw-rw-r—1christinechristine0May2114:35test_one
$touch-atest_one
$ls-ltest_one
-rw-rw-r—1christinechristine0May2114:35test_one
$ls-l—time=atimetest_one
-rw-rw-r—1christinechristine0May2114:55test_one
$
Intheprecedingexample,noticethatbyusingonlythe ls-lcommand,theaccesstime
doesnotdisplay.Thisisbecausethemodificationtimeisshownbydefault.Toseeafile’s
access time, you need to add an additional parameter, —time=atime. After we add that
parameterintheprecedingexample,thefile’salteredaccesstimeisdisplayed.
CreatingemptyfilesandalteringfiletimestampsisnotsomethingyouwilldoonaLinux
systemdaily.However,copyingfilesisanactionyouwilldooftenwhileusingtheshell.
Copyingfiles
Copyingfilesanddirectoriesfromonelocationinthefilesystemtoanotherisacommon
practiceforsystemadministrators.Thecpcommandprovidesthisfeature.
Initsmostbasicform,the cpcommandusestwoparameters—thesourceobjectandthe
destinationobject:cpsourcedestination.
When both the source and destination parameters are filenames, the cp command
copies the source file to a new destination file. The new file acts like a brand new file,
withanupdatedmodificationtime:
$cptest_onetest_two
$ls-ltest_*
-rw-rw-r—1christinechristine0May2114:35test_one
-rw-rw-r—1christinechristine0May2115:15test_two
$
Thenewfile test_twoshowsadifferentmodificationtimethanthe test_onefile.Ifthe
destinationfilealreadyexists,the cpcommandmaynotpromptyoutothisfact.Itisbest
toaddthe-ioptiontoforcetheshelltoaskwhetheryouwanttooverwriteafile:
$ls-ltest_*
-rw-rw-r—1christinechristine0May2114:35test_one
-rw-rw-r—1christinechristine0May2115:15test_two
$
$cp-itest_onetest_two
cp:overwrite‘test_two’?n
$
Ifyoudon’tanswer y,thefilecopydoesnotproceed.Youcanalsocopyafileintoapreexistingdirectory:
$cp-itest_one/home/christine/Documents/
$
$ls-l/home/christine/Documents
total0
-rw-rw-r—1christinechristine0May2115:25test_one
$
The new file is now under the Documents subdirectory, using the same filename as the
original.
Note
Theprecedingexampleusesatrailingforwardslash(/)onthedestinationdirectory
name.UsingtheslashindicatesDocumentsisadirectoryandnotafile.Thisis
helpfulforclaritypurposesandisimportantwhencopyingsinglefiles.Iftheforward
slashisnotusedandthesubdirectory/home/christine/Documentsdoesnotexist,
problemscanoccur.Inthiscase,attemptingtocopyasinglefiletotheDocuments
subdirectorycreatesafilenamedDocumentsinstead,andnoerrormessagesdisplay!
This last example used an absolute directory reference, but you can just as easily use a
relativedirectoryreference:
$cp-itest_oneDocuments/
cp:overwrite‘Documents/test_one’?y
$
$ls-lDocuments
total0
-rw-rw-r—1christinechristine0May2115:28test_one
$
Earlier in this chapter, you read about the special symbols that can be used in relative
directoryreferences.Oneofthem,thesingledot(.),isgreattousewiththe cpcommand.
Remember that the single dot represents your present working directory. If you need to
copyafilewithalongsourceobjectnametoyourpresentworkingdirectory,thesingle
dotcansimplifythetask:
$cp-i/etc/NetworkManager/NetworkManager.conf.
$
$ls-lNetworkManager.conf
-rw-r—r—1christinechristine76May2115:55NetworkManager.conf
$
It’s hard to see that single dot! If you look closely, you’ll see it at the end of the first
example code line. Using the single dot symbol is much easier than typing a full
destinationobjectname,whenyouhavelongsourceobjectnames.
Tip
Therearemanymorecpcommandparametersthanthosedescribedhere.Remember
thatyoucanseeallthedifferentavailableparametersavailableforthecpcommand,
bytypingmancp.
The-Rparameterisapowerful cpcommandoption.Itallowsyoutorecursivelycopythe
contentsofanentiredirectoryinonecommand:
$ls-Fd*Scripts
Scripts/
$ls-lScripts/
total25
-rwxrw-r—1christinechristine929Apr208:23file_mod.sh
-rwxrw-r—1christinechristine254Jan214:18SGID_search.sh
-rwxrw-r—1christinechristine243Jan213:42SUID_search.sh
$
$cp-RScripts/Mod_Scripts
$ls-Fd*Scripts
Mod_Scripts/Scripts/
$ls-lMod_Scripts
total25
-rwxrw-r—1christinechristine929May2116:16file_mod.sh
-rwxrw-r—1christinechristine254May2116:16SGID_search.sh
-rwxrw-r—1christinechristine243May2116:16SUID_search.sh
$
ThedirectoryMod_Scriptsdidnotexistpriortothe cp-Rcommand.Itwascreatedwith
the cp -R command, and the entire Scripts directory’s contents were copied into it.
Noticethatallthefilesinthenew Mod_Scriptsdirectoryhavenewdatesassociatedwith
them.NowMod_ScriptsisacompletecopyoftheScriptsdirectory.
Note
Intheprecedingexample,theoptions-Fdwereaddedtothelscommand.Youread
aboutthe-Foptionearlierinthischapter.However,the-doptionmaybenewtoyou.
The-doptionlistsadirectory’sinformationbutnotitscontents.
Youcanalsousewildcardmetacharactersinyourcpcommands:
$cp*scriptMod_Scripts/
$ls-lMod_Scripts
total26
-rwxrw-r—1christinechristine929May2116:16file_mod.sh
-rwxrw-r—1christinechristine54May2116:27my_script
-rwxrw-r—1christinechristine254May2116:16SGID_search.sh
-rwxrw-r—1christinechristine243May2116:16SUID_search.sh
$
ThiscommandcopiedanyfilesthatendedwithscripttoMod_Scripts.Inthiscase,only
onefileneededtobecopied:my_script.
Whencopyingfiles,anothershellfeaturecanhelpyoubesidesthesingledotandwildcard
metacharacters.Itiscalledtabauto-complete.
Usingtabauto-complete
Whenworkingatthecommandline,youcaneasilymistypeacommand,directoryname,
or filename. In fact, the longer a directory reference or filename, the greater the chance
thatyouwillmistypeit.
Thisiswheretabauto-completecanbealifesaver.Tabauto-completeallowsyoutostart
typingafilenameordirectorynameandthenpressthetabkeytohavetheshellcomplete
itforyou:
$lsreally*
really_ridiculously_long_file_name
$
$cpreally_ridiculously_long_file_nameMod_Scripts/
ls-lMod_Scripts
total26
-rwxrw-r—1christinechristine929May2116:16file_mod.sh
-rwxrw-r—1christinechristine54May2116:27my_script
-rw-rw-r—1christinechristine0May2117:08
really_ridiculously_long_file_name
-rwxrw-r—1christinechristine254May2116:16SGID_search.sh
-rwxrw-r—1christinechristine243May2116:16SUID_search.sh
$
Intheprecedingexample,wetypedthecommandcpreallyandpressedthetabkey,and
theshellauto-completedtherestofthefilename!Ofcourse,thedestinationdirectoryhad
to be typed, but still tab auto-complete saved the command from several potential
typographicalerrors.
Thetricktousingtabauto-completeistogivetheshellenoughfilenamecharacterssoit
candistinguishthedesiredfilefromotherfiles.Forexample,ifanotherfilenamestarted
with really, pressing the tab key would not auto-complete the filename. Instead, you
wouldhearabeep.Ifthishappens,youcanpressthetabkeyagain,andtheshellshows
youallthefilenamesstartingwithreally.Thisfeatureallowsyoutoseewhatneedstobe
typedfortabauto-completetoworkproperly.
Linkingfiles
LinkingfilesisagreatoptionavailableintheLinuxfilesystem.Ifyouneedtomaintain
two(ormore)copiesofthesamefileonthesystem,insteadofhavingseparatephysical
copies,youcanuseonephysicalcopyandmultiplevirtualcopies,calledlinks.Alinkisa
placeholderinadirectorythatpointstothereallocationofthefile.Twotypesoffilelinks
areavailableinLinux:
Asymboliclink
Ahardlink
A symbolic link is simply a physical file that points to another file somewhere in the
virtual directory structure. The two symbolically linked together files do not share the
samecontents.
Tocreateasymboliclinktoafile,theoriginalfilemustpre-exist.Wecanthenusethe ln
commandwiththe-soptiontocreatethesymboliclink:
$ls-ldata_file
-rw-rw-r—1christinechristine1092May2117:27data_file
$
$ln-sdata_filesl_data_file
$
$ls-l*data_file
-rw-rw-r—1christinechristine1092May2117:27data_file
lrwxrwxrwx1christinechristine9May2117:29sl_data_file->
data_file
$
In the preceding example, notice that the name of the symbolic link, sl_data_file, is
listedsecondinthe lncommand.The —>symboldisplayedafterthesymboliclinkfile’s
longlistingshowsthatitissymbolicallylinkedtothefiledata_file.
Alsonotethesymboliclink’sfilesizeversusthedatafile’sfilesize.Thesymboliclink,
sl_data_file, is only 9 bytes, whereas the data_file is 1092 bytes. This is because
sl_data_file is only pointing to data_file. They do not share contents and are two
physicallyseparatefiles.
Another way to tell that these linked files are separate physical files is by viewing their
inodenumber.Theinodenumberofafileordirectoryisauniqueidentificationnumber
thatthekernelassignstoeachobjectinthefilesystem.Toviewafileordirectory’sinode
number,addthe-iparametertothelscommand:
$ls-i*data_file
296890data_file296891sl_data_file
$
Theexampleshowsthatthedatafile’sinodenumberis 296890,whilethe sl_data_file
inodenumberisdifferent.Itis296891.Thus,theyaredifferentfiles.
Ahardlinkcreatesaseparatevirtualfilethatcontainsinformationabouttheoriginalfile
andwheretolocateit.However,theyarephysicallythesamefile.Whenyoureferencethe
hardlinkfile,it’sjustasifyou’rereferencingtheoriginalfile.Tocreateahardlink,again
the original file must pre-exist, except that this time no parameter is needed on the ln
command:
$ls-lcode_file
-rw-rw-r—1christinechristine189May2117:56code_file
$
$lncode_filehl_code_file
$
$ls-li*code_file
296892-rw-rw-r—2christinechristine189May2117:56
code_file
296892-rw-rw-r—2christinechristine189May2117:56
hl_code_file
$
Intheprecedingexample,weusedthe ls-licommandtoshowboththeinodenumbers
and a long listing for the *code_files. Notice that both files, which are hard linked
together,sharethenameinodenumber.Thisisbecausetheyarephysicallythesamefile.
Alsonoticethatthelinkcount(thethirditeminthelisting)nowshowsthatbothfileshave
twolinks.Inaddition,theirfilesizeisexactlythesamesizeaswell.
Note
Youcanonlycreateahardlinkbetweenfilesonthesamephysicalmedium.Tocreate
alinkbetweenfilesunderseparatephysicalmediums,youmustuseasymboliclink.
Be careful when copying linked files. If you use the cp command to copy a file that’s
linkedtoanothersourcefile,allyou’redoingismakinganothercopyofthesourcefile.
Thiscanquicklygetconfusing.Insteadofcopyingthelinkedfile,youcancreateanother
link to the original file. You can have many links to the same file with no problems.
However,youalsodon’twanttocreatesoftlinkstoothersoft-linkedfiles.Thiscreatesa
chainoflinksthatcanbeconfusing—andeasilybroken—causingallsortsofproblems.
Youmayfindsymbolicandhardlinksdifficultconcepts.Fortunately,renamingfilesinthe
nextsectionisagreatdealeasiertounderstand.
Renamingfiles
IntheLinuxworld,renamingfilesiscalledmovingfiles.The mvcommandisavailableto
movebothfilesanddirectoriestoanotherlocationoranewname:
$ls-lif?ll
296730-rw-rw-r—1christinechristine0May2113:44fall
296717-rw-rw-r—1christinechristine0May2113:44fell
294561-rw-rw-r—1christinechristine0May2113:44fill
296742-rw-rw-r—1christinechristine0May2113:44full
$
$mvfallfzll
$
$ls-lif?ll
296717-rw-rw-r—1christinechristine0May2113:44fell
294561-rw-rw-r—1christinechristine0May2113:44fill
296742-rw-rw-r—1christinechristine0May2113:44full
296730-rw-rw-r—1christinechristine0May2113:44fzll
$
Notice that moving the file changed the name from fall to fzll, but it kept the same
inodenumberandtimestampvalue.Thisisbecausemvaffectsonlyafile’sname.
Youcanalsousemvtochangeafile’slocation:
$ls-li/home/christine/fzll
296730-rw-rw-r—1christinechristine0May2113:44
/home/christine/fzll
$
$ls-li/home/christine/Pictures/
total0
$mvfzllPictures/
$
$ls-li/home/christine/Pictures/
total0
296730-rw-rw-r—1christinechristine0May2113:44fzll
$
$ls-li/home/christine/fzll
ls:cannotaccess/home/christine/fzll:Nosuchfileordirectory
$
In the preceding example, we moved the file fzll from /home/christine to
/home/christine/Picturesusingthe mvcommand.Again,therewerenochangestothe
file’sinodenumberortimestampvalue.
Tip
Likethecpcommand,youcanusethe-ioptiononthemvcommand.Thus,youare
askedbeforethecommandattemptstooverwriteanypre-existingfiles.
The only change was to the file’s location. The fzll file no longer exists in
/home/christine, because a copy of it was not left in its original location, as the cp
commandwouldhavedone.
Youcanusethemvcommandtomoveafile’slocationandrenameit,allinoneeasystep:
$ls-liPictures/fzll
296730-rw-rw-r—1christinechristine0May2113:44
Pictures/fzll
$
$mv/home/christine/Pictures/fzll/home/christine/fall
$
$ls-li/home/christine/fall
296730-rw-rw-r—1christinechristine0May2113:44
/home/christine/fall
$
$ls-li/home/christine/Pictures/fzll
ls:cannotaccess/home/christine/Pictures/fzll:
Nosuchfileordirectory
For this example, we moved the file fzll from a subdirectory, Pictures, to the home
directory, /home/christine,andrenameditto fall.Neitherthetimestampvaluenorthe
inodenumberchanged.Onlythelocationandnamewerealtered.
Youcanalsousethemvcommandtomoveentiredirectoriesandtheircontents:
$ls-liMod_Scripts
total26
296886-rwxrw-r—1christinechristine929May2116:16
file_mod.sh
296887-rwxrw-r—1christinechristine54May2116:27
my_script
296885-rwxrw-r—1christinechristine254May2116:16
SGID_search.sh
296884-rwxrw-r—1christinechristine243May2116:16
SUID_search.sh
$
$mvMod_ScriptsOld_Scripts
$
$ls-liMod_Scripts
ls:cannotaccessMod_Scripts:Nosuchfileordirectory
$
$ls-liOld_Scripts
total26
296886-rwxrw-r—1christinechristine929May2116:16
file_mod.sh
296887-rwxrw-r—1christinechristine54May2116:27
my_script
296885-rwxrw-r—1christinechristine254May2116:16
SGID_search.sh
296884-rwxrw-r—1christinechristine243May2116:16
SUID_search.sh
$
Thedirectory’sentirecontentsareunchanged.Theonlythingthatchangesisthenameof
thedirectory.
Afteryouknowhowtorename…err…movefileswiththe mvcommand,yourealizehow
simpleitistoaccomplish.Anothereasy,butpotentiallydangerous,taskisdeletingfiles.
Deletingfiles
Most likely at some point you’ll want to be able to delete existing files. Whether it’s to
cleanupafilesystemortoremoveasoftwarepackage,youalwayshaveopportunitiesto
deletefiles.
IntheLinuxworld,deletingiscalledremoving.Thecommandtoremovefilesinthebash
shellisrm.Thebasicformofthermcommandissimple:
$rm-ifall
rm:removeregularemptyfile‘fall’?y
$
$ls-lfall
ls:cannotaccessfall:Nosuchfileordirectory
$
Noticethatthe-icommandparameterpromptsyoutomakesurethatyou’reseriousabout
removingthefile.Theshellhasnorecyclebinortrashcan.Afteryouremoveafile,it’s
gone forever. Therefore, a good habit is to always tack on the -i parameter to the rm
command.
Youcanalsousewildcardmetacharacterstoremovegroupsoffiles.However,again,use
that-ioptiontoprotectyourself:
$rm-if?ll
rm:removeregularemptyfile‘fell’?y
rm:removeregularemptyfile‘fill’?y
rm:removeregularemptyfile‘full’?y
$
$ls-lf?ll
ls:cannotaccessf?ll:Nosuchfileordirectory
$
Oneotherfeatureofthermcommand,ifyou’reremovinglotsoffilesanddon’twanttobe
botheredwiththeprompt,istousethe-fparametertoforcetheremoval.Justbecareful!
ManagingDirectories
Linux has a few commands that work for both files and directories (such as the cp
command),andsomethatworkonlyfordirectories.Tocreateanewdirectory,youneedto
use a specific command, which is covered in this section. Removing directories can get
interesting,sothatiscoveredinthissectionaswell.
Creatingdirectories
CreatinganewdirectoryinLinuxiseasy—justusethemkdircommand:
$mkdirNew_Dir
$ls-ldNew_Dir
drwxrwxr-x2christinechristine4096May2209:48New_Dir
$
The system creates a new directory named New_Dir. Notice in the new directory’s long
listingthatthedirectory’srecordbeginswitha d.Thisindicatesthat New_Dirisnotafile,
butadirectory.
Youcancreatedirectoriesandsubdirectoriesin“bulk”ifneeded.However,ifyouattempt
thiswithjustthemkdircommand,yougetthefollowingerrormessage:
$mkdirNew_Dir/Sub_Dir/Under_Dir
mkdir:cannotcreatedirectory‘New_Dir/Sub_Dir/Under_Dir’:
Nosuchfileordirectory
$
Tocreateseveraldirectoriesandsubdirectoriesatthesametime,youneedtoaddthe -p
parameter:
$mkdir-pNew_Dir/Sub_Dir/Under_Dir
$
$ls-RNew_Dir
New_Dir:
Sub_Dir
New_Dir/Sub_Dir:
Under_Dir
New_Dir/Sub_Dir/Under_Dir:
$
The-poptiononthemkdircommandmakesanymissingparentdirectoriesasneeded.A
parent directory is a directory that contains other directories at the next level down the
directorytree.
Ofcourse,afteryoumakesomething,youneedtoknowhowtodeleteit.Thisisespecially
usefulifyoucreatedadirectoryinthewronglocation.
Deletingdirectories
Removingdirectoriescanbetricky,andforgoodreason.Therearelotsofopportunities
forbadthingstohappenwhenyoustartdeletingdirectories.Theshelltriestoprotectus
from accidental catastrophes as much as possible. The basic command for removing a
directoryisrmdir:
$touchNew_Dir/my_file
$ls-liNew_Dir/
total0
294561-rw-rw-r—1christinechristine0May2209:52my_file
$
$rmdirNew_Dir
rmdir:failedtoremove‘New_Dir’:Directorynotempty
$
Bydefault,the rmdircommandworksonlyforremovingemptydirectories.Becausewe
createdafile,my_file,intheNew_Dirdirectory,thermdircommandrefusestoremoveit.
Tofixthis,wemustremovethefilefirst.Thenwecanusethermdircommandonthenow
emptydirectory:
$rm-iNew_Dir/my_file
rm:removeregularemptyfile‘New_Dir/my_file’?y
$
$rmdirNew_Dir
$
$ls-ldNew_Dir
ls:cannotaccessNew_Dir:Nosuchfileordirectory
Thermdirhasno -ioptiontoaskifyouwanttoremovethedirectory.Thisisonereason
itishelpfulthatrmdirremovesonlyemptydirectories.
You can also use the rm command on entire non-empty directories. Using the -roption
allowsthecommandtodescendintothedirectory,removethefiles,andthenremovethe
directoryitself:
$ls-lMy_Dir
total0
-rw-rw-r—1christinechristine0May2210:02another_file
$
$rm-riMy_Dir
rm:descendintodirectory‘My_Dir’?y
rm:removeregularemptyfile‘My_Dir/another_file’?y
rm:removedirectory‘My_Dir’?y
$
$ls-lMy_Dir
ls:cannotaccessMy_Dir:Nosuchfileordirectory
$
Thisalsoworksfordescendingintomultiplesubdirectoriesandisespeciallyusefulwhen
youhavelotsofdirectoriesandfilestodelete:
$ls-FRSmall_Dir
Small_Dir:
a_fileb_filec_fileTeeny_Dir/Tiny_Dir/
Small_Dir/Teeny_Dir:
e_file
Small_Dir/Tiny_Dir:
d_file
$
$rm-irSmall_Dir
rm:descendintodirectory‘Small_Dir’?y
rm:removeregularemptyfile‘Small_Dir/a_file’?y
rm:descendintodirectory‘Small_Dir/Tiny_Dir’?y
rm:removeregularemptyfile‘Small_Dir/Tiny_Dir/d_file’?y
rm:removedirectory‘Small_Dir/Tiny_Dir’?y
rm:descendintodirectory‘Small_Dir/Teeny_Dir’?y
rm:removeregularemptyfile‘Small_Dir/Teeny_Dir/e_file’?y
rm:removedirectory‘Small_Dir/Teeny_Dir’?y
rm:removeregularemptyfile‘Small_Dir/c_file’?y
rm:removeregularemptyfile‘Small_Dir/b_file’?y
rm:removedirectory‘Small_Dir’?y
$
$ls-FRSmall_Dir
ls:cannotaccessSmall_Dir:Nosuchfileordirectory
$
Althoughthisworks,it’ssomewhatawkward.Noticethatyoustillmustverifyeachand
everyfilethatgetsremoved.Foradirectorywithlotsoffilesandsubdirectories,thiscan
becometedious.
Note
Forthermcommand,the-rparameterandthe-Rparameterworkexactlythesame.
Whenusedwiththermcommand,the-Rparameteralsorecursivelytraversesthrough
thedirectoryremovingfiles.Itisunusualforashellcommandtohavedifferentcased
parameterswiththesamefunction.
Theultimatesolutionforthrowingcautiontothewindandremovinganentiredirectory,
contentsandall,isthermcommandwithboththe-rand-fparameters:
$treeSmall_Dir
Small_Dir
├a_file
├b_file
├c_file
├Teeny_Dir
|└e_file
└Tiny_Dir
└d_file
2directories,5files
$
$rm-rfSmall_Dir
$
$treeSmall_Dir
Small_Dir[erroropeningdir]
0directories,0files
$
The rm-rfcommandgivesnowarningsandnofanfare.This,ofcourse,isanextremely
dangeroustooltohave,especiallyifhavesuperuserprivileges.Useitsparingly,andonly
aftertriplecheckingtomakesurethatyou’redoingexactlywhatyouwanttodo!
Note
Noticeintheprecedingexamplethatweusedthetreeutility.Itnicelydisplays
directories,subdirectories,andtheirfiles.It’sausefulutilitywhenyouneedto
understandadirectorystructure,especiallybeforeremovingit.Thisutilitymaynot
beinstalledbydefaultinyourLinuxdistribution.SeeChapter9forlearningabout
installingsoftware
In the last few sections, you looked at managing both files and directories. So far we
coveredeverythingyouneedtoknowaboutfiles,exceptforhowtopeekinsideofthem.
ViewingFileContents
Youcanuseseveralcommandsforlookinginsidefileswithouthavingtopulloutatext
editorutility(seeChapter10).Thissectiondemonstratesafewofthecommandsyouhave
availabletohelpyouexaminefiles.
Viewingthefiletype
Beforeyougochargingofftryingtodisplayafile,trytogetahandleonwhattypeoffile
itis.Ifyoutrytodisplayabinaryfile,yougetlotsofgibberishonyourmonitorandmay
evenlockupyourterminalemulator.
Thefilecommandisahandylittleutilitytohavearound.Itcanpeekinsideofafileand
determinejustwhatkindoffileitis:
$filemy_file
my_file:ASCIItext
$
Thefileintheprecedingexampleisa textfile.The filecommanddeterminednotonly
thatthefilecontainstextbutalsothecharactercodeformatofthetextfile,ASCII.
Thisfollowingexampleshowsafilethatissimplyadirectory.Thus,the filecommand
givesyouanothermethodtodistinguishadirectory:
$fileNew_Dir
New_Dir:directory
$
Thisthird filecommandexampleshowsafile,whichisasymboliclink.Notethatthe
filecommandeventellsyoutowhichfileitissymbolicallylinked:
$filesl_data_file
sl_data_file:symboliclinkto‘data_file’
$
Thefollowingexampleshowswhatthe filecommandreturnsforascriptfile.Although
thefileisASCIItext,becauseit’sascriptfile,youcanexecute(run)itonthesystem:
$filemy_script
my_script:Bourne-Againshellscript,ASCIItextexecutable
$
The final example is a binary executable program. The file command determines the
platformthattheprogramwascompiledforandwhattypesoflibrariesitrequires.Thisis
an especially handy feature if you have a binary executable program from an unknown
source:
$file/bin/ls
/bin/ls:ELF64-bitLSBexecutable,x86-64,version1(SYSV),
dynamicallylinked(usessharedlibs),forGNU/Linux2.6.24,
[…]
$
Nowthatyouknowaquickmethodforviewingafile’stype,youcanstartdisplayingand
viewingfiles.
Viewingthewholefile
Ifyouhavealargetextfileonyourhands,youmaywanttobeabletoseewhat’sinsideof
it.Linuxhasthreedifferentcommandsthatcanhelpyouhere.
Usingthecatcommand
Thecatcommandisahandytoolfordisplayingallthedatainsideatextfile:
$cattest1
hello
Thisisatestfile.
Thatwe’llusetotestthecatcommand.
$
Nothingtooexciting,justthecontentsofthetextfile.However,the catcommandhasa
fewparametersthatcanhelpyouout.
The-nparameternumbersallthelinesforyou:
$cat-ntest1
1hello
2
3Thisisatestfile.
4
5
6Thatwe’llusetotestthecatcommand.
$
That feature will come in handy when you’re examining scripts. If you just want to
numberthelinesthathavetextinthem,the-bparameterisforyou:
$cat-btest1
1hello
2Thisisatestfile.
3Thatwe’llusetotestthecatcommand.
$
Finally,ifyoudon’twanttabcharacterstoappear,usethe-Tparameter:
$cat-Ttest1
hello
Thisisatestfile.
Thatwe’lluseto∧Itestthecatcommand.
$
The-Tparameterreplacesanytabsinthetextwiththe∧Icharactercombination.
For large files, the cat command can be somewhat annoying. The text in the file just
quickly scrolls off the display without stopping. Fortunately, we have a simple way to
solvethisproblem.
Usingthemorecommand
Themaindrawbackofthe catcommandisthatyoucan’tcontrolwhat’shappeningafter
you start it. To solve that problem, developers created the more command. The more
commanddisplaysatextfile,butstopsafteritdisplayseachpageofdata.Wetypedthe
commandmore/etc/bash.bashrctoproducethesamplemorescreenshowninFigure3.3.
Figure3.3Usingthemorecommandtodisplayatextfile
Notice at the bottom of the screen in Figure3.3 that the more command displays a tag
showingthatyou’restillinthe moreapplicationandhowfaralong(56%)inthetextfile
youare.Thisisthepromptforthemorecommand.
Themorecommandisapagerutility.Rememberfromearlierinthischapterapagerutility
displays selected bash manual pages when you use the man command. Similarly to
navigating through the man pages, you can use more to navigate through a text file by
pressingthespacebaroryoucangoforwardlinebylineusingtheEnterkey.Whenyou
arefinishednavigatingthroughthefileusingmore,presstheqkeytoquit.
The more command allows some rudimentary movement through the text file. For more
advancedfeatures,trythelesscommand.
Usingthelesscommand
Fromitsname,itsoundslikeitshouldn’tbeasadvancedasthemorecommand.However,
the less command name is actually a play on words and is an advanced version of the
more command (the less command name comes from the phrase “less is more”). It
providesseveralveryhandyfeaturesforscrollingbothforwardandbackwardthrougha
textfile,aswellassomeprettyadvancedsearchingcapabilities.
The lesscommandcanalsodisplayafile’scontentsbeforeitfinishesreadingtheentire
file.Thecatandmorecommandscannotdothis.
Thelesscommandoperatesmuchthesameasthe morecommand,displayingonescreen
oftextfromafileatatime.Itsupportsthesamecommandsetasthemorecommand,plus
manymoreoptions.
Tip
Toseealltheoptionsavailableforthelesscommand,viewitsmanpagesbytyping
manless.Youcandothesameforthemorecommandtoseethereferencematerial
concerningitsvariousoptionsaswell.
Onesetoffeaturesisthatthe lesscommandrecognizestheupanddownarrowkeysas
wellasthePageUpandPageDownkeys(assumingthatyou’reusingaproperlydefined
terminal).Thisgivesyoufullcontrolwhenviewingafile.
Viewingpartsofafile
Oftenthedatayouwanttoviewislocatedeitherrightatthetoporburiedatthebottomof
atextfile.Iftheinformationisatthetopofalargefile,youstillneedtowaitforthe cat
or more commands to load the entire file before you can view it. If the information is
locatedatthebottomofafile(suchasalogfile),youneedtowadethroughthousandsof
lines of text just to get to the last few entries. Fortunately, Linux has specialized
commandstosolvebothoftheseproblems.
Usingthetailcommand
The tailcommanddisplaysthelastlinesinafile(thefile’s“tail”).Bydefault,itshows
thelast10linesinthefile.
Fortheseexamples,wecreatedatextfilecontaining20textlines.Itisdisplayedherein
itsentiretyusingthecatcommand:
$catlog_file
line1
line2
line3
line4
line5
HelloWorld-line6
line7
line8
line9
line10
line11
Helloagain-line12
line13
line14
line15
Sweet-line16
line17
line18
line19
Lastline-line20
$
Nowthatyouhaveseentheentiretextfile,youcanseetheeffectofusing tailtoview
thefile’slast10lines:
$taillog_file
line11
Helloagain-line12
line13
line14
line15
Sweet-line16
line17
line18
line19
Lastline-line20
$
Youcanchangethenumberoflinesshownusing tailbyincludingthe -nparameter.In
thisexample,onlythelasttwolinesofthefilearedisplayed,byadding -n2tothe tail
command:
$tail-n2log_file
line19
Lastline-line20
$
The -f parameter is a pretty cool feature of the tail command. It allows you to peek
insideafileasthefileisbeingusedbyotherprocesses.The tailcommandstaysactive
andcontinuestodisplaynewlinesastheyappearinthetextfile.Thisisagreatwayto
monitorthesystemlogfilesinreal-timemode.
Usingtheheadcommand
The head command does what you’d expect; it displays a file’s first group of lines (the
file’s“head”).Bydefault,itdisplaysthefirst10linesoftext:
$headlog_file
line1
line2
line3
line4
line5
HelloWorld-line6
line7
line8
line9
line10
$
Similartothe tailcommand,the headcommandsupportsthe -nparametersoyoucan
alterwhat’sdisplayed.Bothcommandsalsoallowyoutosimplytypeadashalongwith
thenumberoflinestodisplay,asshownhere:
$head-5log_file
line1
line2
line3
line4
line5
$
Usuallythebeginningofafiledoesn’tchange,sotheheadcommanddoesn’tsupportthefparameterfeatureasthetailcommanddoes.Theheadcommandisahandywaytojust
peekatthebeginningofafile.
Summary
ThischaptercoveredthebasicsofworkingwiththeLinuxfilesystemfromashellprompt.
We began with a discussion of the bash shell and showed you how to interact with the
shell.Thecommandlineinterface(CLI)usesapromptstringtoindicatewhenit’sready
foryoutoentercommands.
Theshellprovidesawealthofutilitiesyoucanusetocreateandmanipulatefiles.Before
youstartplayingwithfiles,youshouldunderstandhowLinuxstoresthem.Thischapter
discussedthebasicsoftheLinuxvirtualdirectoryandshowedyouhowLinuxreferences
storage media devices. After describing the Linux filesystem, the chapter walked you
throughusingthecdcommandtomovearoundthevirtualdirectory.
Aftershowingyouhowtogettoadirectory,thechapterdemonstratedhowtousethe ls
commandtolistthefilesandsubdirectories.Lotsofparameterscancustomizetheoutput
ofthe lscommand.Youcanobtaininformationonfilesanddirectoriesbyusingthe ls
command.
The touch command is useful for creating empty files and for changing the access or
modificationtimesonanexistingfile.Thechapteralsodiscussedusingthe cpcommand
tocopyexistingfilesfromonelocationtoanother.Itwalkedyouthroughtheprocessof
linkingfilesinsteadofcopyingthem,providinganeasywaytohavethesamefileintwo
locationswithoutmakingaseparatecopy.Thelncommandprovidesthislinkingability.
Next, you learned how to rename files (called moving) in Linux using the mv command
and saw how to delete files (called removing) using the rm command. This chapter also
showedyouhowtoperformthesametaskswithdirectories,usingthe mkdirand rmdir
commands.
Finally, the chapter closed with a discussion on viewing the contents of files. The cat,
more,and lesscommandsprovideeasymethodsforviewingtheentirecontentsofafile,
whilethe tailand headcommandsaregreatforpeekinginsideafiletojustseeasmall
portionofit.
The next chapter continues the discussion on bash shell commands. We’ll look at more
advanced administrator commands that come in handy as you administer your Linux
system.
Chapter4
MorebashShellCommands
InThisChapter
1. Managingprocesses
2. Gettingdiskstatistics
3. Mountingnewdisks
4. Sortingdata
5. Archivingdata
Chapter3coveredthebasicsofwalkingthroughtheLinuxfilesystemandworking
withfilesanddirectories.Fileanddirectorymanagementisamajorfeatureofthe
Linuxshell;however,weshouldlookatsomeotherthingsbeforewestartourscript
programming.ThischapterdigsintotheLinuxsystemmanagementcommands,
showingyouhowtopeekinsideyourLinuxsystemusingcommandlinecommands.
Afterthat,weshowyouafewhandycommandsthatyoucanusetoworkwithdata
filesonthesystem.
MonitoringPrograms
OneofthetoughestjobsofbeingaLinuxsystemadministratoriskeepingtrackofwhat’s
running on the system — especially now, when graphical desktops take a handful of
programsjusttoproduceasingledesktop.Youalwayshavelotsofprogramsrunningon
thesystem.
Fortunately,afewcommandlinetoolsareavailabletohelpmakelifeeasierforyou.This
sectioncoversafewofthebasictoolsyouneedtoknowhowtousetomanageprograms
onyourLinuxsystem.
Peekingattheprocesses
When a program runs on the system, it’s referred to as a process. To examine these
processes,youneedtobecomefamiliarwiththe pscommand,theSwissArmyknifeof
utilities.Itcanproducelotsofinformationaboutalltheprogramsrunningonyoursystem.
Unfortunately, with this robustness comes complexity — in the form of numerous
parameters—makingthe pscommandprobablyoneofthemostdifficultcommandsto
master. Most system administrators find a subset of these parameters that provide the
informationtheywant,andtheystickwithusingonlythose.
That said, however, the basic ps command doesn’t really provide all that much
information:
$ps
PIDTTYTIMECMD
3081pts/000:00:00bash
3209pts/000:00:00ps
$
Nottooexciting.Bydefault,thepscommandshowsonlytheprocessesthatbelongtothe
current user and that are running on the current terminal. In this case, we had only our
bash shell running (remember, the shell is just another program running on the system)
and,ofcourse,thepscommanditself.
The basic output shows the process ID (PID) of the programs, the terminal (TTY) that
theyarerunningfrom,andtheCPUtimetheprocesshasused.
Note
Thetrickyfeatureofthepscommand(andthepartthatmakesitsocomplicated)is
thatatonetimethereweretwoversionsofit.Eachversionhaditsownsetof
commandlineparameterscontrollingwhatinformationitdisplayedandhow.
Recently,Linuxdevelopershavecombinedthetwopscommandformatsintoasingle
psprogram(andofcourseaddedtheirowntouches).
The GNU ps command that’s used in Linux systems supports three different types of
commandlineparameters:
Unix-styleparameters,whichareprecededbyadash
BSD-styleparameters,whicharenotprecededbyadash
GNUlongparameters,whichareprecededbyadoubledash
Thefollowingsectionsexaminethethreedifferentparametertypesandshowexamplesof
howtheywork.
Unix-styleparameters
TheUnix-styleparametersoriginatedwiththeoriginalpscommandthatranontheAT&T
UnixsystemsinventedbyBellLabs.Table4.1showstheseparameters.
Table4.1ThepsCommandUnixParameters
Parameter
-Ggrplist
Description
Showsallprocesses
Showstheoppositeofthespecifiedparameters
Showsallprocessesexceptsessionheadersandprocesseswithouta
terminal
Showsallprocessesexceptsessionheaders
Showsallprocesses
Showsprocessescontainedinthelistcmdlist
ShowsprocesseswithagroupIDlistedingrplist
-U
userlist
Showsprocessesownedbyauseridlistedinuserlist
-ggrplist
-ppidlist
Showsprocessesbysessionorbygroupidcontainedingrplist
ShowsprocesseswithPIDsinthelistpidlist
-s
sesslist
ShowsprocesseswithsessionIDinthelistsesslist
-tttylist
ShowsprocesseswithterminalIDinthelistttylist
-A
-N
-a
-d
-e
-Ccmslist
-u
userlist
Showsprocessesbyeffectiveuseridinthelistuserlist
Usesextrafulloutput
-Oformat Displaysspecificcolumnsinthelistformat,alongwiththedefaultcolumns
-M
Displayssecurityinformationabouttheprocess
-c
Showsadditionalschedulerinformationabouttheprocess
-f
Displaysafullformatlisting
-j
Showsjobinformation
-l
Displaysalonglisting
-oformat
Displaysonlyspecificcolumnslistedinformat
-y
Preventsdisplayofprocessflags
-Z
Displaysthesecuritycontextinformation
-H
Displaysprocessesinahierarchicalformat(showingparentprocesses)
-F
-n
namelist
DefinesthevaluestodisplayintheWCHANcolumn
-w
Useswideoutputformat,forunlimitedwidthdisplays
Showsprocessthreads
Displaystheversionofps
-L
-V
That’salotofparameters,andtherearestillmore!Thekeytousingthe pscommandis
not to memorize all the available parameters — only those you find most useful. Most
Linuxsystemadministratorshavetheirownsetsofparametersthattheyuseforextracting
pertinentinformation.Forexample,ifyouneedtoseeeverythingrunningonthesystem,
use the -ef parameter combination (the ps command lets you combine parameters like
this):
$ps-ef
UIDPIDPPIDCSTIMETTYTIMECMD
root10011:29?00:00:01init[5]
root20011:29?00:00:00[kthreadd]
root32011:29?00:00:00[migration/0]
root42011:29?00:00:00[ksoftirqd/0]
root52011:29?00:00:00[watchdog/0]
root62011:29?00:00:00[events/0]
root72011:29?00:00:00[khelper]
root472011:29?00:00:00[kblockd/0]
root482011:29?00:00:00[kacpid]
6823491011:30?00:00:00hald
root30781981012:00?00:00:00sshd:rich[priv]
rich30803078012:00?00:00:00sshd:rich@pts/0
rich30813080012:00pts/000:00:00-bash
rich44453081313:48pts/000:00:00ps-ef
$
Quiteafewlineshavebeencutfromtheoutputtosavespace,butyoucanseethatlotsof
processes are running on a Linux system. This example uses two parameters: the -e
parameter, which shows all the processes running on the system, and the -f parameter,
whichexpandstheoutputtoshowafewusefulcolumnsofinformation:
UID:Theuserresponsibleforlaunchingtheprocess
PID:TheprocessIDoftheprocess
PPID:ThePIDoftheparentprocess(ifaprocessisstartedbyanotherprocess)
C:Processorutilizationoverthelifetimeoftheprocess
STIME:Thesystemtimewhentheprocessstarted
TTY:Theterminaldevicefromwhichtheprocesswaslaunched
TIME:ThecumulativeCPUtimerequiredtoruntheprocess
CMD:Thenameoftheprogramthatwasstarted
This produces a reasonable amount of information, which is what many system
administrators want to see. For even more information, you can use the -l parameter,
whichproducesthelongformatoutput:
$ps-l
FSUIDPIDPPIDCPRINIADDRSZWCHANTTYTIMECMD
0S500308130800800-1173waitpts/000:00:00bash
0R500446330811800-1116-pts/000:00:00ps
$
Noticetheextracolumnsthatappearwhenyouusethe-lparameter:
F:Systemflagsassignedtotheprocessbythekernel
S:Thestateoftheprocess(O=runningonprocessor;S=sleeping;R=runnable,
waitingtorun;Z=zombie,processterminatedbutparentnotavailable;T=process
stopped)
PRI:Thepriorityoftheprocess(highernumbersmeanlowerpriority)
NI:Thenicevalue,whichisusedfordeterminingpriorities
ADDR:Thememoryaddressoftheprocess
SZ:Approximateamountofswapspacerequirediftheprocesswasswappedout
WCHAN:Addressofthekernelfunctionwheretheprocessissleeping
BSD-styleparameters
Now that you’ve seen the Unix parameters, let’s look at the BSD-style parameters. The
BerkeleySoftwareDistribution(BSD)wasaversionofUnixdevelopedat(ofcourse)the
University of California, Berkeley. It had many subtle differences from the AT&T Unix
system,thussparkingmanyUnixwarsovertheyears.Table4.2showstheBSDversionof
thepscommandparameters.
Table4.2ThepsCommandBSDParameters
Parameter
Description
x
Showsallprocessesassociatedwiththisterminal
Showsallprocessesassociatedwithanyterminal
Showsallprocessesincludingsessionheaders
Showsonlyrunningprocesses
Showsallprocesses,eventhosewithoutaterminaldeviceassigned
U
userlist
Showsprocessesownedbyauseridlistedinuserlist
T
a
g
r
ppidlist
ShowsprocesseswithaPIDlistedinpidlist
tttylist
Showsprocessesassociatedwithaterminallistedinttylist
Oformat Listsspecificcolumnsinformattodisplayalongwiththestandardcolumns
X
Displaysdataintheregisterformat
Z
Includessecurityinformationintheoutput
j
Showsjobinformation
l
Usesthelongformat
oformat
Displaysonlycolumnsspecifiedinformat
s
Usesthesignalformat
u
Usestheuser-orientedformat
v
Usesthevirtualmemoryformat
N
namelist
DefinesthevaluestouseintheWCHANcolumn
Oorder
Definestheorderinwhichtodisplaytheinformationcolumns
Sumsnumericalinformation,suchasCPUandmemoryusage,forchild
processesintotheparentprocess
Displaysthetruecommandname(thenameoftheprogramusedtostartthe
process)
Displaysanyenvironmentvariablesusedbythecommand
Displaysprocessesinahierarchicalformat,showingwhichprocessesstarted
whichprocesses
Preventsdisplayoftheheaderinformation
Definesthecolumn(s)touseforsortingtheoutput
UsesnumericvaluesforuserandgroupIDs,alongwithWCHANinformation
Produceswideoutputforwiderterminals
Displaysthreadsasiftheywereprocesses
Displaysthreadsaftertheirprocesses
Listsallformatspecifiers
Displaystheversionofps
S
c
e
f
h
ksort
n
w
H
m
L
V
Asyoucansee,theUnixandBSDtypesofparametershavelotsofoverlap.Mostofthe
informationyoucangetfromoneyoucanalsogetfromtheother.Mostofthetime,you
choose a parameter type based on which format you’re more comfortable with (for
example,ifyouwereusedtoaBSDenvironmentbeforeusingLinux).
When you use the BSD-style parameters, the ps command automatically changes the
outputtosimulatetheBSDformat.Here’sanexampleusingthelparameter:
$psl
FUIDPIDPPIDPRINIVSZRSSWCHANSTATTTYTIMECOMMAND
05003081308020046921432waitSspts/00:00-bash
0500510430812004468844-R+pts/00:00psl
$
Notice that while many of the output columns are the same as when we used the Unixstyleparameters,somedifferentonesappearaswell:
VSZ:Thesizeinkilobytesoftheprocessinmemory
RSS:Thephysicalmemorythataprocesshasusedthatisn’tswappedout
STAT:Atwo-characterstatecoderepresentingthecurrentprocessstate
Many system administrators like the BSD-style l parameter because it produces a more
detailed state code for processes (the STAT column). The two-character code more
precisely defines exactly what’s happening with the process than the single-character
Unix-styleoutput.
ThefirstcharacterusesthesamevaluesastheUnix-styleSoutputcolumn,showingwhen
a process is sleeping, running, or waiting. The second character further defines the
process’sstatus:
<:Theprocessisrunningathighpriority.
N:Theprocessisrunningatlowpriority.
L:Theprocesshaspageslockedinmemory.
s:Theprocessisasessionleader.
l:Theprocessismulti-threaded.
+:Theprocessisrunningintheforeground.
From the simple example shown previously, you can see that the bash command is
sleeping, but it is a session leader (it’s the main process in my session), whereas the ps
commandwasrunningintheforegroundonthesystem.
TheGNUlongparameters
Finally,theGNUdevelopersputtheirowntouchesonthenew,improvedpscommandby
addingafewmoreoptionstotheparametermix.SomeoftheGNUlongparameterscopy
existingUnix-orBSD-styleparameters,whileothersprovidenewfeatures.Table4.3lists
theavailableGNUlongparameters.
Table4.3ThepsCommandGNUParameters
Parameter
—deselect
—Groupgrplist
—Useruserlist
—groupgrplist
—pidpidlist
—ppidpidlist
—sidsidlist
—ttyttylist
—useruserlist
—formatformat
—context
—colsn
—columnsn
—cumulative
—forest
—headers
—no-headers
—linesn
—rowsn
—sortorder
—widthn
—help
—info
—version
Description
Showsallprocessesexceptthoselistedinthecommandline
ShowsprocesseswhosegroupIDislistedingrplist
ShowsprocesseswhoseuserIDislistedinuserlist
ShowsprocesseswhoseeffectivegroupIDislistedingrplist
ShowsprocesseswhoseprocessIDislistedinpidlist
ShowsprocesseswhoseparentprocessIDislistedinpidlist
ShowsprocesseswhosesessionIDislistedinsidlist
ShowsprocesseswhoseterminaldeviceIDislistedinttylist
ShowsprocesseswhoseeffectiveuserIDislistedinuserlist
Displaysonlycolumnsspecifiedintheformat
Displaysadditionalsecurityinformation
Setsscreenwidthtoncolumns
Setsscreenwidthtoncolumns
Includesstoppedchildprocessinformation
Displaysprocessesinahierarchicallistingshowingparentprocesses
Repeatscolumnheadersoneachpageofoutput
Preventsdisplayofcolumnheaders
Setsthescreenheighttonlines
Setsthescreenheighttonrows
Definesthecolumn(s)touseforsortingtheoutput
Setsthescreenwidthtoncolumns
Displaysthehelpinformation
Displaysdebugginginformation
Displaystheversionofthepsprogram
You can combine GNU long parameters with either Unix- or BSD-style parameters to
really customize your display. One cool feature of GNU long parameters that we really
likeisthe —forestparameter.Itdisplaysthehierarchicalprocessinformation,butusing
ASCIIcharacterstodrawcutecharts:
1981?00:00:00sshd
3078?00:00:00\_sshd
3080?00:00:00\_sshd
3081pts/000:00:00\_bash
16676pts/000:00:00\_ps
Thisformatmakestracingchildandparentprocessesasnap!
Real-timeprocessmonitoring
Thepscommandisgreatforgleaninginformationaboutprocessesrunningonthesystem,
but it has one drawback. The ps command can display information only for a specific
pointintime.Ifyou’retryingtofindtrendsaboutprocessesthatarefrequentlyswappedin
andoutofmemory,it’shardtodothatwiththepscommand.
Instead, the top command can solve this problem. The top command displays process
informationsimilarlytothe pscommand,butitdoesitinreal-timemode.Figure4.1isa
snapshotofthetopcommandinaction.
Figure4.1Theoutputofthetopcommandwhileitisrunning
Thefirstsectionoftheoutputshowsgeneralsysteminformation.Thefirstlineshowsthe
current time, how long the system has been up, the number of users logged in, and the
loadaverageonthesystem.
Theloadaverageappearsasthreenumbers:the1-minute,5-minute,and15-minuteload
averages. The higher the values, the more load the system is experiencing. It’s not
uncommon for the 1-minute load value to be high for short bursts of activity. If the 15minuteloadvalueishigh,yoursystemmaybeintrouble.
Note
ThetrickinLinuxsystemadministrationisdefiningwhatexactlyahighloadaverage
valueis.Thisvaluedependsonwhat’snormallyrunningonyoursystemandthe
hardwareconfiguration.What’shighforonesystemmightbenormalforanother.
Usually,ifyourloadaveragesstartgettingover2,thingsaregettingbusyonyour
system.
The second line shows general process information (called tasks in top): how many
processes are running, sleeping, stopped, and zombie (have finished but their parent
processhasn’tresponded).
The next line shows general CPU information. The top display breaks down the CPU
utilization into several categories depending on the owner of the process (user versus
systemprocesses)andthestateoftheprocesses(running,idle,orwaiting).
Following that are two lines that detail the status of the system memory. The first line
showsthestatusofthephysicalmemoryinthesystem,howmuchtotalmemorythereis,
howmuchiscurrentlybeingused,andhowmuchisfree.Thesecondmemorylineshows
the status of the swap memory area in the system (if any is installed), with the same
information.
Finally, the next section shows a detailed list of the currently running processes, with
someinformationcolumnsthatshouldlookfamiliarfromthepscommandoutput:
PID:TheprocessIDoftheprocess
USER:Theusernameoftheowneroftheprocess
PR:Thepriorityoftheprocess
NI:Thenicevalueoftheprocess
VIRT:Thetotalamountofvirtualmemoryusedbytheprocess
RES:Theamountofphysicalmemorytheprocessisusing
SHR:Theamountofmemorytheprocessissharingwithotherprocesses
S:Theprocessstatus(D=interruptiblesleep,R=running,S=sleeping,T=tracedor
stopped,orZ=zombie)
%CPU:TheshareofCPUtimethattheprocessisusing
%MEM:Theshareofavailablephysicalmemorytheprocessisusing
TIME+:ThetotalCPUtimetheprocesshasusedsincestarting
COMMAND:Thecommandlinenameoftheprocess(programstarted)
Bydefault,whenyoustart top, it sorts the processes based on the %CPUvalue.Youcan
changethesortorderbyusingoneofseveralinteractivecommandswhile topisrunning.
Eachinteractivecommandisasinglecharacterthatyoucanpresswhile top isrunning
andchangesthebehavioroftheprogram.Pressingfallowsyoutoselectthefieldtouseto
sorttheoutput,andpressingdallowsyoutochangethepollinginterval.Pressqtoexitthe
topdisplay.
You have lots of control over the output of the top command. Using this tool, you can
oftenfindoffendingprocessesthathavetakenoveryoursystem.Ofcourse,afteryoufind
one,thenextjobistostopit,whichbringsustothenexttopic.
Stoppingprocesses
Acrucialpartofbeingasystemadministratorisknowingwhenandhowtostopaprocess.
Sometimes,aprocessgetshungupandneedsagentlenudgetoeithergetgoingagainor
stop. Other times, a process runs away with the CPU and refuses to give it up. In both
cases,youneedacommandthatallowsyoutocontrolaprocess.LinuxfollowstheUnix
methodofinterprocesscommunication.
In Linux, processes communicate with each other using signals. A process signal is a
predefined message that processes recognize and may choose to ignore or act on. The
developers program how a process handles signals. Most well-written applications have
theabilitytoreceiveandactonthestandardUnixprocesssignals.Table4.4showsthese
signals.
Table4.4LinuxProcessSignals
Signal
1
2
3
9
11
15
17
18
19
Name
Description
HUP
Hangsup
INT
Interrupts
QUIT
Stopsrunning
KILL
Unconditionallyterminates
SEGV
Producessegmentviolation
TERM
Terminatesifpossible
STOP
Stopsunconditionally,butdoesn’tterminate
TSTP Stopsorpauses,butcontinuestoruninbackground
CONT
ResumesexecutionafterSTOPorTSTP
TwocommandsavailableinLinuxallowyoutosendprocesssignalstorunningprocesses.
Thekillcommand
The kill command allows you to send signals to processes based on their process ID
(PID). By default, the kill command sends a TERM signal to all the PIDs listed on the
commandline.Unfortunately,youcanonlyusetheprocessPIDinsteadofitscommand
name,makingthekillcommanddifficulttousesometimes.
Tosendaprocesssignal,youmusteitherbetheowneroftheprocessorbeloggedinas
therootuser.
$kill3940
-bash:kill:(3940)-Operationnotpermitted
$
The TERM signal tells the process to kindly stop running. Unfortunately, if you have a
runawayprocess,mostlikelyitignorestherequest.Whenyouneedtogetforceful,the -s
parameterallowsyoutospecifyothersignals(eitherusingtheirnameorsignalnumber).
As you can see from the following example, no output is associated with the kill
command.
#kill-sHUP3940
#
Toseeifthecommandwaseffective,youmustperformanotherpsortopcommandtosee
iftheoffendingprocessstopped.
Thekillallcommand
The killall command is a powerful way to stop processes by using their names rather
thanthePIDnumbers.The killall command allows you to use wildcard characters as
well,makingitaveryusefultoolwhenyouhaveasystemthat’sgoneawry:
#killallhttp*
#
Thisexamplekillsalltheprocessesthatstartwithhttp,suchasthehttpdservicesforthe
Apachewebserver.
Caution
Beextremelycarefulusingthekillallcommandwhenloggedinastherootuser.It’s
easytogetcarriedawaywithwildcardcharactersandaccidentallystopimportant
systemprocesses.Thiscouldleadtoadamagedfilesystem.
MonitoringDiskSpace
Anotherimportanttaskofthesystemadministratoristokeeptrackofthediskusageon
thesystem.Whetheryou’rerunningasimpleLinuxdesktoporalargeLinuxserver,you
needtoknowhowmuchspaceyouhaveforyourapplications.
Some command line commands can help you manage the media environment on your
Linuxsystem.Thissectiondescribesthecorecommandsyou’lllikelyrunintoduringyour
systemadministrationduties.
Mountingmedia
As discussed in Chapter 3, the Linux filesystem combines all media disks into a single
virtualdirectory.Beforeyoucanuseanewmediadiskonyoursystem,youmustplaceit
inthevirtualdirectory.Thistaskiscalledmounting.
In today’s graphical desktop world, most Linux distributions have the ability to
automatically mount specific types of removablemedia. A removable media device is a
mediumthat(obviously)canbeeasilyremovedfromthePC,suchasCD-ROMsandUSB
memorysticks.
If you’re not using a distribution that automatically mounts and unmounts removable
media, you have to do it yourself. This section describes the Linux command line
commandstohelpyoumanageyourremovablemediadevices.
Themountcommand
Oddlyenough,thecommandusedtomountmediaiscalled mount.Bydefault,the mount
commanddisplaysalistofmediadevicescurrentlymountedonthesystem:
$mount
/dev/mapper/VolGroup00-LogVol00on/typeext3(rw)
procon/proctypeproc(rw)
sysfson/systypesysfs(rw)
devptson/dev/ptstypedevpts(rw,gid=5,mode=620)
/dev/sda1on/boottypeext3(rw)
tmpfson/dev/shmtypetmpfs(rw)
noneon/proc/sys/fs/binfmt_misctypebinfmt_misc(rw)
sunrpcon/var/lib/nfs/rpc_pipefstyperpc_pipefs(rw)
/dev/sdb1on/media/disktypevfat
(rw,nosuid,nodev,uhelper=hal,shortname=lower,uid=503)
$
Themountcommandprovidesfourpiecesofinformation:
Thedevicefilenameofthemedia
Themountpointinthevirtualdirectorywherethemediaismounted
Thefilesystemtype
Theaccessstatusofthemountedmedia
ThelastentryintheprecedingexampleisaUSBmemorystickthattheGNOMEdesktop
automaticallymountedatthe /media/diskmountpoint.Thevfatfilesystemtypeshows
thatitwasformattedonaMicrosoftWindowsPC.
Tomanuallymountamediadeviceinthevirtualdirectory,youmustbeloggedinasthe
rootuserorusethesudocommandtorunthecommandastherootuser.Thefollowingis
thebasiccommandformanuallymountingamediadevice:
mount-ttypedevicedirectory
Thetypeparameterdefinesthefilesystemtypeunderwhichthediskwasformatted.Linux
recognizeslotsofdifferentfilesystemtypes.Ifyoushareremovablemediadeviceswith
yourWindowsPCs,youaremostlikelytorunintothesetypes:
vfat:Windowslongfilesystem
ntfs:WindowsadvancedfilesystemusedinWindowsNT,XP,andVista
iso9660:ThestandardCD-ROMfilesystem
MostUSBmemorysticksandfloppiesareformattedusingthevfatfilesystem.Ifyouneed
tomountadataCD,youmustusetheiso9660filesystemtype.
Thenexttwoparametersdefinethelocationofthedevicefileforthemediadeviceandthe
locationinthevirtualdirectoryforthemountpoint.Forexample,tomanuallymountthe
USBmemorystickatdevice /dev/sdb1atlocation /media/disk,youusethefollowing
command:
mount-tvfat/dev/sdb1/media/disk
Afteramediadeviceismountedinthevirtualdirectory,therootuserhasfullaccesstothe
device, but access by other users is restricted. You can control who has access to the
deviceusingdirectorypermissions(discussedinChapter7).
Incaseyouneedtousesomeofthemoreexoticfeaturesofthemountcommand,Table4.5
showstheavailableparameters.
Table4.5ThemountCommandParameters
Parameter
-a
-f
-F
-v
-I
-l
-n
Description
Mountsallfilesystemsspecifiedinthe/etc/fstabfile
Causesthemountcommandtosimulatemountingadevice,butnotactually
mountit
Mountsallfilesystemsatthesametimewhenusedwiththe-aparameter
Explainsallthestepsrequiredtomountthedevice;standsforverbosemode
Tellsyounottouseanyfilesystemhelperfilesunder/sbin/mount
.filesystem
Addsthefilesystemlabelsautomaticallyforext2,ext3,orXFSfilesystems
Mountsthedevicewithoutregisteringitinthe/etc/mstabmounteddevice
file
-pnum
-s
-r
-w
-Llabel
-Uuuid
-O
-o
Forencryptedmounting,readsthepassphrasefromthefiledescriptornum
Ignoresmountoptionsnotsupportedbythefilesystem
Mountsthedeviceasread-only
Mountsthedeviceasread-write(thedefault)
Mountsthedevicewiththespecifiedlabel
Mountsthedevicewiththespecifieduuid
Whenusedwiththe-aparameter,limitsthesetoffilesystemsapplied
Addsspecificoptionstothefilesystem
The -o option allows you to mount the filesystem with a comma-separated list of
additionaloptions.Thesearepopularoptionstouse:
ro:Mountsasread-only
rw:Mountsasread-write
user:Allowsanordinaryusertomountthefilesystem
check=none:Mountsthefilesystemwithoutperforminganintegritycheck
loop:Mountsafile
Theunmountcommand
Toremovearemovablemediadevice,youshouldneverjustremoveitfromthesystem.
Instead,youshouldalwaysunmountitfirst.
Tip
Linuxdoesn’tallowyoutoejectamountedCD.Ifyoueverhavetroubleremovinga
CDfromthedrive,mostlikelyitmeanstheCDisstillmountedinthevirtual
directory.Unmountitfirst,andthentrytoejectit.
Thecommandusedtounmountdevicesis umount (yes, there’s no “n” in the command,
whichgetsconfusingsometimes).Theformatfortheumountcommandisprettysimple:
umount[directory|device]
The umount command gives you the choice of defining the media device by either its
devicelocationoritsmounteddirectoryname.Ifanyprogramhasafileopenonadevice,
thesystemwon’tletyouunmountit.
[root@testboxmnt]#umount/home/rich/mnt
umount:/home/rich/mnt:deviceisbusy
umount:/home/rich/mnt:deviceisbusy
[root@testboxmnt]#cd/home/rich
[root@testboxrich]#umount/home/rich/mnt
[root@testboxrich]#ls-lmnt
total0
[root@testboxrich]#
In this example, the command prompt was still in a directory within the filesystem
structure,sothe umount command couldn’t unmount the image file. After the command
prompt was moved out of the image file filesystem, the umount command successfully
unmountedtheimagefile.
Usingthedfcommand
Sometimes, you need to see how much disk space is available on an individual device.
Thedfcommandallowsyoutoeasilyseewhat’shappeningonallthemounteddisks:
$df
Filesystem1K-blocksUsedAvailableUse%Mountedon
/dev/sda2182510687703964960502445%/
/dev/sda1101086186807718720%/boot
tmpfs11953601195360%/dev/shm
/dev/sdb11274621138921357090%/media/disk
$
Thedfcommandshowseachmountedfilesystemthatcontainsdata.Asyoucanseefrom
themountcommandearlier,somemounteddevicesareusedforinternalsystempurposes.
Thecommanddisplaysthefollowing:
Thedevicelocationofthedevice
Howmany1024-byteblocksofdataitcanhold
Howmany1024-byteblocksareused
Howmany1024-byteblocksareavailable
Theamountofusedspaceasapercentage
Themountpointwherethedeviceismounted
A few different command line parameters are available with the df command, most of
which you’ll never use. One popular parameter is -h, which shows the disk space in
human-readableform,usuallyasanMformegabytesoraGforgigabytes:
$df-h
FilesystemSizeUsedAvailUse%Mountedon
/dev/sdb218G7.4G9.2G45%/
/dev/sda199M19M76M20%/boot
tmpfs117M0117M0%/dev/shm
/dev/sdb1125M112M14M90%/media/disk
$
Nowinsteadofhavingtodecodethoseuglyblocknumbers,allthedisksizesareshown
using “normal” sizes. The df command is invaluable in troubleshooting disk space
problemsonthesystem.
Note
RememberthattheLinuxsystemalwayshasprocessesrunninginthebackground
thathandlefiles.ThevaluesfromthedfcommandreflectwhattheLinuxsystem
thinksarethecurrentvaluesatthatpointintime.It’spossiblethatyouhaveaprocess
runningthathascreatedordeletedafilebuthasnotreleasedthefileyet.Thisvalue
isnotincludedinthefreespacecalculation.
Usingtheducommand
Withthe dfcommand,youcaneasilyseewhenadiskisrunningoutofspace.Thenext
problemforthesystemadministratoristoknowwhattodowhenthathappens.
Anotherusefulcommandtohelpyouistheducommand.Theducommandshowsthedisk
usage for a specific directory (by default, the current directory). This is a quick way to
determineifyouhaveanyobviousdiskhogsonthesystem.
Bydefault,theducommanddisplaysallthefiles,directories,andsubdirectoriesunderthe
currentdirectory,anditshowshowmanydiskblockseachfileordirectorytakes.Fora
standard-sizeddirectory,thiscanbequitealisting.Here’sapartiallistingofusingthe du
command:
$du
484./.gstreamer-0.10
8./Templates
8./Download
8./.ccache/7/0
24./.ccache/7
368./.ccache/a/d
384./.ccache/a
424./.ccache
8./Public
8./.gphpedit/plugins
32./.gphpedit
72./.gconfd
128./.nautilus/metafiles
384./.nautilus
72./.bittorrent/data/metainfo
20./.bittorrent/data/resume
144./.bittorrent/data
152./.bittorrent
8./Videos
8./Music
16./.config/gtk-2.0
40./.config
8./Documents
Thenumberattheleftofeachlineisthenumberofdiskblocksthateachfileordirectory
takes. Notice that the listing starts at the bottom of a directory and works its way up
throughthefilesandsubdirectoriescontainedwithinthedirectory.
The ducommandbyitselfcanbesomewhatuseless.It’snicetobeabletoseehowmuch
diskspaceeachindividualfileanddirectorytakesup,butitcanbemeaninglesswhenyou
havetowadethroughpagesandpagesofinformationbeforeyoufindwhatyou’relooking
for.
Youcanuseafewcommandlineparameterswiththe ducommandtomakethingsalittle
morelegible:
-c:Producesagrandtotalofallthefileslisted
-h:Printssizesinhuman-readableform,usingKforkilobyte,Mformegabyte,and
Gforgigabyte
-s:Summarizeseachargument
The next step for the system administrator is to use some file-handling commands for
manipulatinglargeamountsofdata.That’sexactlywhatthenextsectioncovers.
WorkingwithDataFiles
Whenyouhavealargeamountofdata,handlingtheinformationandmakingitusefulcan
bedifficult.Asyousawwiththeducommandintheprevioussection,it’seasytogetdata
overloadwhenworkingwithsystemcommands.
TheLinuxsystemprovidesseveralcommandlinetoolstohelpyoumanagelargeamounts
of data. This section covers the basic commands that every system administrator — as
wellasanyeverydayLinuxuser—shouldknowhowtousetomaketheirliveseasier.
Sortingdata
The sortcommandisapopularfunctionthatcomesinhandywhenworkingwithlarge
amountsofdata.Thesortcommanddoeswhatitsays:Itsortsdata.
Bydefault,thesortcommandsortsthedatalinesinatextfileusingstandardsortingrules
forthelanguageyouspecifyasthedefaultforthesession.
$catfile1
one
two
three
four
five
$sortfile1
five
four
one
three
two
$
It’sprettysimple,butthingsaren’talwaysaseasyastheyappear.Lookatthisexample:
$catfile2
1
2
100
45
3
10
145
75
$sortfile2
1
10
100
145
2
3
45
75
$
Ifyouwereexpectingthenumberstosortinnumericalorder,youweredisappointed.By
default, the sort command interprets numbers as characters and performs a standard
charactersort,producingoutputthatmightnotbewhatyouwant.Tosolvethisproblem,
use the -n parameter, which tells the sort command to recognize numbers as numbers
insteadofcharactersandtosortthembasedontheirnumericalvalues:
$sort-nfile2
1
2
3
10
45
75
100
145
$
Now, that’s much better! Another common parameter that’s used is -M, the month sort.
Linuxlogfilesusuallycontainatimestampatthebeginningofthelinetoindicatewhen
theeventoccurred:
Sep1307:10:09testboxsmartd[2718]:Device:/dev/sda,opened
Ifyousortafilethatusestimestampdatesusingthedefaultsort,yougetsomethinglike
this:
$sortfile3
Apr
Aug
Dec
Feb
Jan
Jul
Jun
Mar
May
Nov
Oct
Sep
$
It’s not exactly what you wanted. If you use the -M parameter, the sort command
recognizesthethree-charactermonthnomenclatureandsortsappropriately:
$sort-Mfile3
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
$
Table4.6showsotherhandysortparametersyoucanuse.
Table4.6ThesortCommandParameters
Single
DoubleDash
Dash
—ignore-b
leading-blanks
—check=
-C
quiet
-c
-d
-f
-g
-i
-k
-M
-m
-n
-o
-R
-r
-S
-s
-T
—check
—dictionaryorder
—ignore-case
—generalnumeric-sort
—ignorenonprinting
—key=
POS1[,POS2]
—month-sort
—merge
—numericsort
—output=
file
—random-sort
—randomsource=FILE
—reverse
—buffer-size
=SIZE
—stable
—temporarydirection=
Description
Ignoresleadingblankswhensorting
Doesn’tsort,butdoesn’treportifdataisoutofsortorder
Doesn’tsort,butchecksiftheinputdataisalreadysorted,and
reportsifnotsorted
Considersonlyblanksandalphanumericcharacters;doesn’t
considerspecialcharacters
Bydefault,sortorderscapitalizedlettersfirst;ignorescase
Usesgeneralnumericalvaluetosort
Ignoresnonprintablecharactersinthesort
SortsbasedonpositionPOS1,andendsatPOS2ifspecified
Sortsbymonthorderusingthree-charactermonthnames
Mergestwoalreadysorteddatafiles
Sortsbystringnumericalvalue
Writesresultstofilespecified
Sortsbyarandomhashofkeys
Specifiesthefileforrandombytesusedbythe-Rparameter
Reversesthesortorder(descendinginsteadofascending
Specifiestheamountofmemorytouse
Disableslast-resortcomparison
Specifiesalocationtostoretemporaryworkingfiles
DIR
-t
—fieldseparator=
SEP
-u
—unique
Specifiesthecharacterusedtodistinguishkeypositions
Withthe-cparameter,checksforstrictordering;withoutthe-c
parameter,outputsonlythefirstoccurrenceoftwosimilarlines
-z
—zeroterminated
EndsalllineswithaNULLcharacterinsteadofanewline
The -k and -t parameters are handy when sorting data that uses fields, such as the
/etc/passwdfile.Usethe-tparametertospecifythefieldseparatorcharacter,andusethe
-kparametertospecifywhichfieldtosorton.Forexample,tosortthepasswordfilebased
onnumericaluserid,justdothis:
$sort-t‘:’-k3-n/etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
news:x:9:13:news:/etc/news:
uucp:x:10:14:uucp:/var/spool/uucp:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
gopher:x:13:30:gopher:/var/gopher:/sbin/nologin
ftp:x:14:50:FTPUser:/var/ftp:/sbin/nologin
Now the data is perfectly sorted based on the third field, which is the numerical userid
value.
The -n parameter is great for sorting numerical outputs, such as the output of the du
command:
$du-sh*|sort-nr
1008kmrtg-2.9.29.tar.gz
972kbldg1
888kfbs2.pdf
760kPrinttest
680krsync-2.6.6.tar.gz
660kcode
516kfig1001.tiff
496ktest
496kphp-common-4.0.4pl1-6mdk.i586.rpm
448kMesaGLUT-6.5.1.tar.gz
400kplp
Noticethatthe -roptionalsosortsthevaluesindescendingorder,soyoucaneasilysee
whatfilesaretakingupthemostspaceinyourdirectory.
Note
Thepipecommand(|)usedinthisexampleredirectstheoutputoftheducommandto
thesortcommand.That’sdiscussedinmoredetailinChapter11.
Searchingfordata
Often in a large file, you must look for a specific line of data buried somewhere in the
middle of the file. Instead of manually scrolling through the entire file, you can let the
grepcommandsearchforyou.Thecommandlineformatforthegrepcommandis:
grep[options]pattern[file]
The grepcommandsearcheseithertheinputorthefileyouspecifyforlinesthatcontain
charactersthatmatchthespecifiedpattern.Theoutputfrom grepisthelinesthatcontain
thematchingpattern.
Herearetwosimpleexamplesofusingthe grepcommandwiththe file1fileusedinthe
“Sortingdata”section:
$grepthreefile1
three
$greptfile1
two
three
$
The first example searches the file file1 for text matching the pattern three. The grep
commandproducesthelinethatcontainsthematchingpattern.Thenextexamplesearches
the file file1 for the text matching the pattern t. In this case, two lines matched the
specifiedpattern,andbotharedisplayed.
Because of the popularity of the grep command, it has undergone lots of development
changesoveritslifetime.Lotsoffeatureshavebeenaddedtothe grepcommand.Ifyou
lookoverthemanpagesforthegrepcommand,you’llseehowversatileitreallyis.
If you want to reverse the search (output lines that don’t match the pattern), use the -v
parameter:
$grep-vtfile1
one
four
five
$
If you need to find the line numbers where the matching patterns are found, use the -n
parameter:
$grep-ntfile1
2:two
3:three
$
Ifyoujustneedtoseeacountofhowmanylinescontainthematchingpattern,usethe -c
parameter:
$grep-ctfile1
2
$
If you need to specify more than one matching pattern, use the -e parameter to specify
eachindividualpattern:
$grep-et-effile1
two
three
four
five
$
Thisexampleoutputslinesthatcontaineitherthestringtorthestringf.
By default, the grep command uses basic Unix-style regular expressions to match
patterns.AUnix-styleregularexpressionusesspecialcharacterstodefinehowtolookfor
matchingpatterns.
Foramoredetailedexplanationofregularexpressions,seeChapter20.
Here’sasimpleexampleofusingaregularexpressioninagrepsearch:
$grep[tf]file1
two
three
four
five
$
Thesquarebracketsintheregularexpressionindicatethat grepshouldlookformatches
thatcontaineitheratoranfcharacter.Withouttheregularexpression, grepwouldsearch
fortextthatwouldmatchthestringtf.
Theegrepcommandisanoffshootofgrep,whichallowsyoutospecifyPOSIXextended
regular expressions, which contain more characters for specifying the matching pattern
(again, see Chapter 20 for more details). The fgrep command is another version that
allows you to specify matching patterns as a list of fixed-string values, separated by
newlinecharacters.Thisallowsyoutoplacealistofstringsinafileandthenusethatlist
inthefgrepcommandtosearchforthestringsinalargerfile.
Compressingdata
Ifyou’vedoneanyworkintheMicrosoftWindowsworld,nodoubtyou’veusedzipfiles.
It became such a popular feature that Microsoft eventually incorporated it into the
WindowsoperatingsystemstartingwithXP.Theziputilityallowsyoutoeasilycompress
largefiles(bothtextandexecutable)intosmallerfilesthattakeuplessspace.
Linuxcontainsseveralfilecompressionutilities.Althoughthismaysoundgreat,itoften
leads to confusion and chaos when trying to download files. Table 4.7 lists the file
compressionutilitiesavailableforLinux.
Table4.7LinuxFileCompressionUtilities
Utility
File
Extension
bzip2
.bz2
compress
.Z
gzip
.gz
zip
.zip
Description
UsestheBurrows-Wheelerblocksortingtextcompression
algorithmandHuffmancoding
OriginalUnixfilecompressionutility;startingtofadeawayinto
obscurity
TheGNUProject’scompressionutility;usesLempel-Zivcoding
TheUnixversionofthePKZIPprogramforWindows
The compress file compression utility is not often found on Linux systems. If you
downloadafilewitha .Zextension,youcanusuallyinstallthecompresspackage(called
ncompressinmanyLinuxdistributions)usingthesoftwareinstallationmethodsdiscussed
inChapter9andthenuncompressthefilewiththeuncompresscommand.Thegziputility
isthemostpopularcompressiontoolusedinLinux.
ThegzippackageisacreationoftheGNUProject,intheirattempttocreateafreeversion
oftheoriginalUnixcompressutility.Thispackageincludesthesefiles:
gzipforcompressingfiles
gzcatfordisplayingthecontentsofcompressedtextfiles
gunzipforuncompressingfiles
Theseutilitiesworkthesamewayasthebzip2utilities:
$gzipmyprog
$ls-lmy*
-rwxrwxr-x1richrich21972007-09-1311:29myprog.gz
$
The gzipcommandcompressesthefileyouspecifyonthecommandline.Youcanalso
specifymorethanonefilenameorevenusewildcardcharacterstocompressmultiplefiles
atonce:
$gzipmy*
$ls-lmy*
-rwxr—r—1richrich103Sep613:43myprog.c.gz
-rwxr-xr-x1richrich5178Sep613:43myprog.gz
-rwxr—r—1richrich59Sep613:46myscript.gz
-rwxr—r—1richrich60Sep613:44myscript2.gz
$
The gzip command compresses every file in the directory that matches the wildcard
pattern.
Archivingdata
Althoughthe zipcommandworksgreatforcompressingandarchivingdataintoasingle
file, it’s not the standard utility used in the Unix and Linux worlds. By far the most
populararchivingtoolusedinUnixandLinuxisthetarcommand.
The tar command was originally used to write files to a tape device for archiving.
However,itcanalsowritetheoutputtoafile,whichhasbecomeapopularwaytoarchive
datainLinux.
Thefollowingistheformatofthetarcommand:
tarfunction[options]object1object2…
Thefunctionparameterdefineswhatthetarcommandshoulddo,asshowninTable4.8.
Table4.8ThetarCommandFunctions
Function LongName
-A
-c
-d
-r
-t
-u
-x
Description
—
Appendsanexistingtararchivefiletoanotherexistingtar
concatenate
archivefile
—create
Createsanewtararchivefile
Checksthedifferencesbetweenatararchivefileandthe
—diff
filesystem
—delete
Deletesfromanexistingtararchivefile
—append
Appendsfilestotheendofanexistingtararchivefile
—list
Liststhecontentsofanexistingtararchivefile
Appendsfilestoanexistingtararchivefilethatarenewerthana
—update
filewiththesamenameintheexistingarchive
—extract
Extractsfilesfromanexistingarchivefile
Eachfunctionusesoptionstodefineaspecificbehaviorforthetararchivefile.Table4.9
liststhecommonoptionsthatyoucanusewiththetarcommand.
Table4.9ThetarCommandOptions
Option
-Cdir
-ffile
-j
-p
-v
-z
Description
Changestothespecifieddirectory
Outputsresultstofile(ordevice)file
Redirectsoutputtothebzip2commandforcompression
Preservesallfilepermissions
Listsfilesastheyareprocessed
Redirectstheoutputtothegzipcommandforcompression
Theseoptionsareusuallycombinedtocreatethefollowingscenarios.First,youwantto
createanarchivefileusingthiscommand:
tar-cvftest.tartest/test2/
The above command creates an archive file called test.tar containing the contents of
boththetestdirectoryandthetest2directory.Next,thiscommand:
tar-tftest.tar
lists(butdoesn’textract)thecontentsofthetarfiletest.tar.Finally,thiscommand:
tar-xvftest.tar
extractsthecontentsofthetarfile test.tar.Ifthetarfilewascreatedfromadirectory
structure,theentiredirectorystructureisre-createdstartingatthecurrentdirectory.
Asyoucansee,usingthe tarcommandisasimplewaytocreatearchivefilesofentire
directorystructures.Thisisacommonmethodfordistributingsourcecodefilesforopen
sourceapplicationsintheLinuxworld.
Tip
Ifyoudownloadopensourcesoftware,oftenyouseefilenamesthatendin.tgz.
Thesearegzippedtarfiles,whichcanbeextractedusingthecommandtar-zxvf
filename.tgz.
Summary
ThischapterdiscussedsomeofthemoreadvancedbashcommandsusedbyLinuxsystem
administratorsandprogrammers.The psand topcommandsarevitalindeterminingthe
status of the system, allowing you to see what applications are running and how many
resourcestheyareconsuming.
In this day of removable media, another popular topic for system administrators is
mounting storage devices. The mount command allows you to mount a physical storage
device into the Linux virtual directory structure. To remove the device, use the umount
command.
Finally, the chapter discussed various utilities used for handling data. The sort utility
easilysortslargedatafilestohelpyouorganizedata,andthe greputilityallowsyouto
quickly scan through large data files looking for specific information. Several file
compressionutilitiesareavailableinLinux,including gzipandzip.Eachoneallowsyou
tocompresslargefilestohelpsavespaceonyourfilesystem.TheLinux tarutilityisa
popularwaytoarchivedirectorystructuresintoasinglefilethatcaneasilybeportedto
anothersystem.
ThenextchapterdiscussesLinuxshellsandhowtointeractwiththem.Linuxallowsyou
tocommunicatebetweenshells,whichcancomeinhandywhencreatingsubshellsinyour
scripts.
Chapter5
UnderstandingtheShell
InThisChapter
1. InvestigatingShellTypes
2. UnderstandingtheParent/ChildShellRelationship
3. UsingSubshellsCreatively
4. InvestigatingBuilt-inShellCommands
Nowthatyouknowafewshellbasics,suchasreachingtheshellandrudimentary
shellcommands,itistimetoexploretheactualshellprocess.Tounderstandtheshell,
youneedtounderstandafewCLIbasics.
AshellisnotjustaCLI.Itisacomplicatedinteractiverunningprogram.Entering
commandsandusingtheshelltorunscriptscanraisesomeinterestingandconfusing
issues.Understandingtheshellprocessanditsrelationshipshelpsyouresolvethese
issuesoravoidthemaltogether.
Thischaptertakesyouthroughlearningabouttheshellprocess.Youseehow
subshellsarecreatedandtheirrelationshiptotheparentshell.Thevariedcommands
thatcreatechildprocessesareexploredaswellasbuilt-incommands.Youevenread
aboutsomeshelltipsandtrickstotry.
ExploringShellTypes
The shell program that the system starts depends on your user ID configuration. In the
/etc/passwdfile,theuserIDhasitsdefaultshellprogramlistedinfield#7ofitsrecord.
Thedefaultshellprogramisstartedwhenevertheuserlogsintoavirtualconsoleterminal
orstartsaterminalemulatorintheGUI.
In the following example, user christine has the GNU bash shell as her default shell
program:
$cat/etc/passwd
[…]
Christine:x:501:501:ChristineB:/home/Christine:/bin/bash
$
Thebashshellprogramresidesinthe/bindirectory.Alonglistingreveals/bin/bash(the
bashshell)isanexecutableprogram:
$ls-lF/bin/bash
-rwxr-xr-x.1rootroot938832Jul182013/bin/bash*
$
Several other shell programs are on this particular CentOS distribution. They include
tcsh,whichisbasedofftheoriginalCshell:
$ls-lF/bin/tcsh
-rwxr-xr-x.1rootroot387328Feb212013/bin/tcsh*
$
Also,theDebianbasedversionoftheashshell,dash,isincluded:
$ls-lF/bin/dash
-rwxr-xr-x.1rootroot109672Oct172012/bin/dash*
$
Finally,asoftlink(seeChapter3)oftheCshellpointstothetcshshell:
$ls-lF/bin/csh
lrwxrwxrwx.1rootroot4Mar1815:16/bin/csh->tcsh*
$
Eachofthesedifferentshellprogramscouldbesetasauser’sdefaultshell.However,due
tothebashshell’spopularity,it’sraretouseanyothershellasadefaultshell.
Note
AbriefdescriptionofvariousshellswasincludedinChapter1.Youmaybe
interestedinlearningevenmoreaboutshellsotherthantheGNUbashshell.
AdditionalalternativeshellinformationisinChapter23.
Thedefaultinteractiveshellstartswheneverauserlogsintoavirtualconsoleterminalor
starts a terminal emulator in the GUI. However, another default shell, /bin/sh, is the
defaultsystemshell.Thedefaultsystemshellisusedforsystemshellscripts,suchasthose
neededatstartup.
Often,youseeadistributionwithitsdefaultsystemshellsettothebashshellusingasoft
linkasshownhereonthisCentOSdistribution:
$ls-l/bin/sh
lrwxrwxrwx.1rootroot4Mar1815:05/bin/sh->bash
$
However,beawarethatonsomedistributions,thedefaultsystemshellisdifferentthanthe
defaultinteractiveshell,suchasonthisUbuntudistribution:
$cat/etc/passwd
[…]
christine:x:1000:1000:Christine,,,:/home/christine:/bin/bash
$
$ls-l/bin/sh
lrwxrwxrwx1rootroot4Apr2212:33/bin/sh->dash
$
Notethattheuser, christine,hasherdefaultinteractiveshellsetto /bin/bash,thebash
shell.Butthedefaultsystemshell,/bin/sh,issettothedashshell.
Tip
Forbashshellscripts,thesetwodifferentshells,defaultinteractiveshellanddefault
systemshell,cancauseproblems.Besuretoreadabouttheimportantsyntaxneeded
forabashshellscript’sfirstlineinChapter11toavoidtheseissues.
You are not forced to stick with your default interactive shell. You can start any shell
availableonyourdistribution,simplybytypingitsfilename.Forexample,tostartthedash
shell,youcanrunitdirectlybytypingthecommand/bin/dash:
$/bin/dash
$
Itdoesn’tlooklikeanythinghappened,butthedashshellprogramstarted.The$promptis
a CLI prompt for the dash shell. You can leave the dash shell program by typing the
commandexit:
$exit
exit
$
Again, it looks like nothing happened. However, the dash shell program was exited. To
understand this process, the next section explores the relationship between a login shell
programandanewlystartedshellprogram.
ExploringParentandChildShellRelationships
The default interactive shell started when a user logs into a virtual console terminal or
starts a terminal emulator in the GUI is a parentshell. As you have read so far in this
book,aparentshellprocessprovidesaCLIpromptandwaitsforcommandstobeentered.
When the /bin/bash command or the equivalent bash command is entered at the CLI
prompt,anewshellprogramiscreated.Thisisachildshell.AchildshellalsohasaCLI
promptandwaitsforcommandstobeentered.
Because you do not see any relevant messages when you type bash and spawn a child
shell,anothercommandcanhelpbringclarity.The pscommandwascoveredinChapter
4.Usingthiswiththe-foptionbeforeandafterenteringachildshellisuseful:
$ps-f
UIDPIDPPIDCSTIMETTYTIMECMD
50118411840011:50pts/000:00:00-bash
50124291841413:44pts/000:00:00ps-f
$
$bash
$
$ps-f
UIDPIDPPIDCSTIMETTYTIMECMD
50118411840011:50pts/000:00:00-bash
50124301841013:44pts/000:00:00bash
50124442430113:44pts/000:00:00ps-f
$
The first use of ps -f shows two processes. One process has a process ID of 1841
(secondcolumn)andisrunningthebashshellprogram(lastcolumn).Thesecondprocess
(processID2429)istheactualps-fcommandrunning.
Note
Aprocessisarunningprogram.Thebashshellisaprogram,andwhenitruns,itisa
process.Arunningshellissimplyonetypeofprocess.Therefore,whenreading
aboutrunningabashshell,youoftenseetheword“shell”andtheword“process”
usedinterchangeably
Afterthecommandbashisentered,achildshelliscreated.Thesecond ps-fisexecuted
fromwithinthechildshell.Fromthisdisplay,youcanseethattwobashshellprograms
arerunning.Thefirstbashshellprogram,theparentshellprocess,hastheoriginalprocess
ID (PID) of 1841. The second bash shell program, the child shell process, has a PID of
2430.NotethatthechildshellhasaparentprocessID(PPID)of1841,denotingthatthe
parentshellprocessisitsparent.Figure5.1diagramsthisrelationship.
Figure5.1Parentandchildbashshellprocesses
Whenachildshellprocessisspawned,onlysomeoftheparent’senvironmentiscopiedto
thechildshellenvironment.Thiscancauseproblemswithitemssuchasvariables,andit
iscoveredinChapter6.
Achildshellisalsocalledasubshell.Asubshellcanbecreatedfromaparentshell,anda
subshellcanbecreatedfromanothersubshell:
$ps-f
UIDPIDPPIDCSTIMETTYTIMECMD
50118411840011:50pts/000:00:00-bash
50125321841114:22pts/000:00:00ps-f
$
$bash
$
$bash
$
$bash
$
$ps—forest
PIDTTYTIMECMD
1841pts/000:00:00bash
2533pts/000:00:00\_bash
2546pts/000:00:00\_bash
2562pts/000:00:00\_bash
2576pts/000:00:00\_ps
$
Intheprecedingexample,the bash shellcommandwasenteredthreetimes.Effectively,
this created three subshells. The ps —forest command shows the nesting of these
subshells.Figure5.2alsoshowsthissubshellnesting.
Figure5.2Subshellnesting
The ps-fcommandcanbeusefulinsubshellnesting,becauseitdisplayswhoiswhose
parentviathePPIDcolumn:
$ps-f
UIDPIDPPIDCSTIMETTYTIMECMD
50118411840011:50pts/000:00:00-bash
50125331841014:22pts/000:00:00bash
50125462533014:22pts/000:00:00bash
50125622546014:24pts/000:00:00bash
50125852562114:29pts/000:00:00ps-f
$
Thebashshellprogramcanusecommandlineparameterstomodifytheshellstart.Table
5.1liststhecommandlineparametersavailableinbash.
Table5.1ThebashCommandLineParameters
Parameter
Description
-cstring
Readscommandsfromstringandprocessesthem
-i
Startsaninteractiveshell,allowinginputfromtheuser
-l
Actsasifinvokedasaloginshell
-r
Startsarestrictedshell,limitingtheusertothedefaultdirectory
-s
Readscommandsfromthestandardinput
Youcanfindmorehelponthe bashcommandandevenmorecommandlineparameters
bytypingmanbash.Thebash—helpcommandprovidesadditionalassistanceaswell.
Youcangracefullyexitoutofeachsubshellbyenteringtheexitcommand:
$exit
exit
$
$ps—forest
PIDTTYTIMECMD
1841pts/000:00:00bash
2533pts/000:00:00\_bash
2546pts/000:00:00\_bash
2602pts/000:00:00\_ps
$
$exit
exit
$
$exit
exit
$
$ps—forest
PIDTTYTIMECMD
1841pts/000:00:00bash
2604pts/000:00:00\_ps
$
Notonlydoestheexitcommandallowyoutoleavechildsubshells,butyoucanalsolog
out of your current virtual console terminal or terminal emulation software as well. Just
typeexitintheparentshell,andyougracefullyexittheCLI.
Another time a subshell can be created is when you run a shell script. You learn more
aboutthattopicinChapter11.
Also,youcanspawnsubshellswithoutusingthe bashshellcommandorrunningashell
script.Onewayisbyusingaprocesslist.
Lookingatprocesslists
Onasingleline,youcandesignatealistofcommandstoberunoneafteranother.Thisis
donebyenteringacommandlistusingasemicolon(;)betweencommands:
$pwd;ls;cd/etc;pwd;cd;pwd;ls
/home/Christine
DesktopDownloadsMusicPublicVideos
Documentsjunk.datPicturesTemplates
/etc
/home/Christine
DesktopDownloadsMusicPublicVideos
Documentsjunk.datPicturesTemplates
$
Intheprecedingexample,thecommandsallexecutedoneafteranotherwithnoproblems.
However,thisisnotaprocesslist.Foracommandlisttobeconsideredaprocesslist,the
commandsmustbeencasedinparentheses:
$(pwd;ls;cd/etc;pwd;cd;pwd;ls)
/home/Christine
DesktopDownloadsMusicPublicVideos
Documentsjunk.datPicturesTemplates
/etc
/home/Christine
DesktopDownloadsMusicPublicVideos
Documentsjunk.datPicturesTemplates
$
Though the parentheses addition may not appear to be a big difference, they do cause a
verydifferenteffect.Addingparenthesesandturningthecommandlistintoaprocesslist
createdasubshelltoexecutethecommands.
Note
Aprocesslistisacommandgroupingtype.Anothercommandgroupingtypeputsthe
commandsbetweencurlybracketsandendsthecommandlistwithasemicolon(;).
Thesyntaxisasfollows:{command;}.Usingcurlybracketsforcommandgrouping
doesnotcreateasubshellasaprocesslistdoes.
To indicate if a subshell was spawned, a command using an environment variable is
needed here. (Environment variables are covered in detail in Chapter 6). The command
neededis echo$BASH_SUBSHELL.Ifitreturnsa0,thenthereisnosubshell.Ifitreturns1
ormore,thenthereisasubshell.
First,theexampleusingjustacommandlistisexecutedwiththe echo $BASH_SUBSHELL
tackedontotheend:
$pwd;ls;cd/etc;pwd;cd;pwd;ls;echo$BASH_SUBSHELL
/home/Christine
DesktopDownloadsMusicPublicVideos
Documentsjunk.datPicturesTemplates
/etc
/home/Christine
DesktopDownloadsMusicPublicVideos
Documentsjunk.datPicturesTemplates
0
Attheveryendofthecommands’output,youcanseethenumberzero(0)isdisplayed.
Thisindicatesasubshellwasnotcreatedtoexecutethesecommands.
The results are different using a process list. The list is executed with echo
$BASH_SUBSHELLtackedontotheend:
$(pwd;ls;cd/etc;pwd;cd;pwd;ls;echo$BASH_SUBSHELL)
/home/Christine
DesktopDownloadsMusicPublicVideos
Documentsjunk.datPicturesTemplates
/etc
/home/Christine
DesktopDownloadsMusicPublicVideos
Documentsjunk.datPicturesTemplates
1
In this case, the number one (1) displayed at the output’s end. This indicates a subshell
wasindeedcreatedandusedforexecutingthesecommands.
Thus, a processlist is a command grouping enclosed with parentheses, which creates a
subshell to execute the command(s). You can even create a grandchild subshell by
embeddingparentheseswithinaprocesslist:
$(pwd;echo$BASH_SUBSHELL)
/home/Christine
1
$(pwd;(echo$BASH_SUBSHELL))
/home/Christine
2
Noticeinthefirstprocesslist,thenumberone(1)isdisplayedindicatingachildsubshell
asyouwouldexpect.Howeverintheexample’ssecondprocesslist,additionalparentheses
were added around the echo $BASH_SUBSHELL command. These additional parentheses
causedagrandchildsubshelltobecreatedforthecommand’sexecution.Thus,anumber
two(2)wasdisplayedindicatingasubshellwithinasubshell.
Subshells are often used for multi-processing in shell scripts. However, entering into a
subshell is an expensive method and can significantly slow down processing. Subshell
issues exist also for an interactive CLI shell session. It is not truly multi-processing,
becausetheterminalgetstiedupwiththesubshell’sI/O.
Creativelyusingsubshells
At the interactive shell CLI, you have more productive ways to use subshells. Process
lists,co-processes,andpipes(coveredinChapter11)usesubshells.Theyallcanbeused
effectivelywithintheinteractiveshell.
One productive subshell method in the interactive shell uses background mode. Before
discussing how to use background mode and subshells together, you need to understand
backgroundmodeitself.
Investigatingbackgroundmode
Runningacommandinbackgroundmodeallowsthecommandtobeprocessedandfrees
up your CLI for other use. A classic command to demonstrate background mode is the
sleepcommand.
The sleepcommandacceptsasaparameterthenumberofsecondsyouwanttheprocess
to wait (sleep). This command is often used to introduce pauses in shell scripts. The
commandsleep10causesthesessiontopausefor10secondsandthenreturnashellCLI
prompt:
$sleep10
$
Toputacommandintobackgroundmode,the &characteristackedontoitsend.Putting
the sleep command into background mode allows a little investigation with the ps
command:
$sleep3000&
[1]2396
$ps-f
UIDPIDPPIDCSTIMETTYTIMECMD
christi+23382337010:13pts/900:00:00-bash
christi+23962338010:17pts/900:00:00sleep3000
christi+23972338010:17pts/900:00:00ps-f
$
The sleepcommandwastoldtosleepfor 3000seconds(50minutes)inthebackground
(&).Whenitwasputintothebackground,twoinformationalitemsweredisplayedbefore
the shell CLI prompt was returned. The first informational item is the background job’s
number (1) displayed in brackets. The second item is the background job’s process ID
(2396).
The ps command was used to display the various processes. Notice that the sleep 3000
commandislisted.AlsonotethatitsprocessID(PID)inthesecondcolumnisthesame
PIDdisplayedwhenthecommandwentintothebackground,2396.
Inadditiontothepscommand,youcanusethejobscommandtodisplaybackgroundjob
information.Thejobscommanddisplaysanyuser’sprocesses(jobs)currentlyrunningin
backgroundmode:
$jobs
[1]+Runningsleep3000&
$
Thejobscommandshowsthejobnumber(1)inbrackets.Italsodisplaysthejob’scurrent
status(running)aswellasthecommanditself,(sleep3000&).
Youcanseeevenmoreinformationbyusingthe -l(lowercaseL)parameteronthe jobs
command. The -l parameter displays the command’s PID in addition to the other
information:
$jobs-l
[1]+2396Runningsleep3000&
$
Whenthebackgroundjobisfinished,itscompletionstatusisdisplayed:
[1]+Donesleep3000&
$
Tip
Beawarethatabackgroundjob’scompletionstatuswon’tnecessarilywaittilla
convenienttimetodisplayitself.Don’tletitsurpriseyouwhenajob’scompletion
statusjustsuddenlyappearsonyourscreen.
Backgroundmodeisveryhandy.Anditprovidesamethodforcreatingusefulsubshellsat
theCLI.
Puttingprocesslistsintothebackground
As stated earlier, a process list is a command or series of commands executed within a
subshell. Using a process list including sleep commands and displaying the
BASH_SUBSHELLvariableoperatesasyouwouldexpect:
$(sleep2;echo$BASH_SUBSHELL;sleep2)
1
$
In the preceding example, a two-second pause occurs, the number one (1) is displayed
indicating a single subshell level (child subshell), and then another two-second pause
occursbeforethepromptreturns.Nothingtoodramatichere.
Putting the same process list into background mode can cause a slightly different effect
withcommandoutput:
$(sleep2;echo$BASH_SUBSHELL;sleep2)&
[2]2401
$1
[2]+Done(sleep2;echo$BASH_SUBSHELL;sleep2)
$
PuttingtheprocesslistintothebackgroundcausesajobnumberandprocessIDtoappear,
and the prompt returns. However, the odd event is that the displayed number one (1),
indicatingasingle-levelsubshell,isdisplayedbytheprompt!Don’tletthisconfuseyou.
SimplypresstheEnterkey,andyougetanotherpromptback.
Usingaprocesslistinbackgroundmodeisonecreativemethodforusingsubshellsatthe
CLI.Youcandolargeamountsofprocessingwithinasubshellandnothaveyourterminal
tiedupwiththesubshell’sI/O.
Of course, the process list of sleepand echo commands are just for example purposes.
Creating backup files with tar (see Chapter 4) is a more practical example of using
backgroundprocesslistseffectively:
$(tar-cfRich.tar/home/rich;tar-cfMy.tar/home/christine)&
[3]2423
$
Puttingaprocesslistinbackgroundmodeisnottheonlywaytousesubshellscreativelyat
theCLI.Co-processingisanothermethod.
Lookingatco-processing
Co-processingdoestwothingsatthesametime.Itspawnsasubshellinbackgroundmode
andexecutesacommandwithinthatsubshell.
To perform co-processing, the coproc command is used along with the command to be
executedinthesubshell:
$coprocsleep10
[1]2544
$
Co-processing performs almost identically to putting a command in background mode,
exceptforthefactthatitcreatesasubshell.You’llnoticethatwhenthe coproccommand
and its parameters were entered, a background job was started. The background job
number(1)andprocessID(2544)weredisplayedonthescreen.
Thejobscommandallowsyoutodisplaytheco-processingstatus:
$jobs
[1]+RunningcoprocCOPROCsleep10&
$
From the preceding example, you can see the background command executing in the
subshellis coprocCOPROCsleep10.The COPROCisanamegiventotheprocessbythe
coproc command. You can set the name yourself by using extended syntax for the
command:
$coprocMy_Job{sleep10;}
[1]2570
$
$jobs
[1]+RunningcoprocMy_Job{sleep10;}&
$
Byusingtheextendedsyntax,theco-processingnamewassetto My_Job.Becarefulhere,
becausetheextendedsyntaxisalittletricky.Youhavetomakesurethataspaceappears
afterthefirstcurlybracket({)andbeforethestartofyourcommand.Also,youhaveto
makesurethecommandendswithasemicolon(;).Andyouhavetoensurethataspace
appearsafterthesemicolonandbeforetheclosingcurlybracket(}).
Note
Co-processingallowsyoutogetveryfancyandsend/receiveinformationtothe
processrunninginthesubshell.Theonlytimeyouneedtonameaco-processis
whenyouhavemultipleco-processesrunning,andyouneedtocommunicatewith
themall.Otherwise,justletthecoproccommandsetthenametothedefault,COPROC.
You can be really clever and combine co-processing with process lists creating nested
subshells.Justtypeyourprocesslistandputthecommandcoprocinfrontofit:
$coproc(sleep10;sleep2)
[1]2574
$
$jobs
[1]+RunningcoprocCOPROC(sleep10;sleep2)&
$
$ps—forest
PIDTTYTIMECMD
2483pts/1200:00:00bash
2574pts/1200:00:00\_bash
2575pts/1200:00:00|\_sleep
2576pts/1200:00:00\_ps
$
Just remember that spawning a subshell can be expensive and slow. Creating nested
subshellsisevenmoreso!
Using subshells can provide flexibility at the command line as well as convenience.
Understanding their behavior is important to obtaining this flexibility and convenience.
Command behavior is also important to understand. In the next section, the behavior
differencesbetweenbuilt-inandexternalcommandsareexplored.
UnderstandingShellBuilt-InCommands
While learning about the GNU bash shell, you likely have heard the term built-in
command. It is important to understand both shell built-in and non-built-in (external)
commands.Built-incommandsandnon-built-incommandsoperateverydifferently.
Lookingatexternalcommands
Anexternalcommand,sometimescalledafilesystemcommand,isaprogramthatexists
outsideofthebashshell.Theyarenotbuiltintotheshellprogram.Anexternalcommand
programistypicallylocatedin/bin,/usr/bin,/sbin,or/usr/sbin.
The ps command is an external command. You can find its filename by using both the
whichandthetypecommands:
$whichps
/bin/ps
$
$type-aps
psis/bin/ps
$
$ls-l/bin/ps
-rwxr-xr-x1rootroot93232Jan618:32/bin/ps
$
Whenever an external command is executed, a child process is created. This action is
termedforking.Conveniently,theexternalcommand psdisplaysitscurrentparentaswell
asitsownforkedchildprocesses:
$ps-f
UIDPIDPPIDCSTIMETTYTIMECMD
christi+27432742017:09pts/900:00:00-bash
christi+28012743017:16pts/900:00:00ps-f
$
Because it is an external command, when the ps command executes, a child process is
created.Inthiscase,the pscommand’sPIDis 2801andtheparentPIDis 2743.Thebash
shellprocess,whichistheparent,hasaPIDof2743.Figure5.3illustratestheforkingthat
occurswhenanexternalcommandisexecuted.
Figure5.3Externalcommandforking
Wheneveraprocessmustfork,ittakestimeandefforttosetupthenewchildprocess’s
environment.Thus,externalcommandscanbealittleexpensive.
Note
Ifyouforkachildprocessorcreateasubshell,youcanstillcommunicatewithitvia
signaling,whichisextremelyhelpfulinboththecommandlineandinwritingshell
scripts.Signalingallowsprocesscommunicationviasignals.Signalsandsignaling
arecoveredinChapter16.
Whenusingabuilt-incommand,noforkingisrequired.Therefore,built-incommandsare
lessexpensive.
Lookingatbuilt-incommands
Built-incommandsaredifferentinthattheydonotneedachildprocesstoexecute.They
werecompiledintotheshellandthusarepartoftheshell’stoolkit.Noexternalprogram
fileexiststorunthem.
Boththe cdand exitcommandsarebuiltintothebashshell.Youcantellacommandis
built-inbyusingthetypecommand:
$typecd
cdisashellbuiltin
$
$typeexit
exitisashellbuiltin
$
Becausetheydonotneedtoforkachildprocesstoexecuteoropenaprogramfile,built-in
commands are faster and more efficient. A list of GNU bash shell built-in commands is
providedinAppendixA.
Be aware that some commands have multiple flavors. For example, both echo and pwd
haveabuilt-incommandflavoraswellasanexternalcommandflavor.Theseflavorsare
slightly different. To see multiple flavors for commands, use the -a option on the type
command:
$type-aecho
echoisashellbuiltin
echois/bin/echo
$
$whichecho
/bin/echo
$
$type-apwd
pwdisashellbuiltin
pwdis/bin/pwd
$
$whichpwd
/bin/pwd
$
Usingthe type-acommandshowsbothtypesforeachofthetwocommands.Notethat
thewhichcommandshowsonlytheexternalcommandfile.
Tip
Tousetheexternalcommandforacommandthathasmultipleflavors,directly
referencethefile.Forexample,tousethepwdexternalcommand,type/bin/pwd.
Usingthehistorycommand
A useful built-in command is the history command. The bash shell keeps track of the
commandsyouhaveused.Youcanrecallthesecommandsandevenreusethem.
Toseearecentlyusedcommandslist,justtypethehistorycommandwithnooptions:
$history
1ps-f
2pwd
3ls
4coproc(sleep10;sleep2)
5jobs
6ps—forest
7ls
8ps-f
9pwd
10ls-l/bin/ps
11history
12cd/etc
13pwd
14ls
15cd
16typepwd
17whichpwd
18typeecho
19whichecho
20type-apwd
21type-aecho
22pwd
23history
Inthisexample,onlythelast23commandsareshown.Typically,thelast1,000commands
arekeptinhistory.Thatislotsofcommands!
Tip
Youcansetthenumberofcommandstokeepinthebashhistory.Todoso,youneed
tomodifyanenvironmentvariablecalledHISTSIZE(seeChapter6).
You can recall and reuse the last command in your history list. This can save time and
typing.Torecallandreuseyourlastcommand,type!!andpresstheEnterkey:
$ps—forest
PIDTTYTIMECMD
2089pts/000:00:00bash
2744pts/000:00:00\_ps
$
$!!
ps—forest
PIDTTYTIMECMD
2089pts/000:00:00bash
2745pts/000:00:00\_ps
$
When!!wasentered,thebashshellfirstdisplayedthecommanditwasrecallingfromthe
shell’shistory.Afterthecommandwasdisplayed,itwasexecuted.
Commandhistoryiskeptinthehidden .bash_historyfile,whichislocatedintheuser’s
homedirectory.Becarefulhere.Thebashcommandhistoryisstoredinmemoryandthen
writtenoutintothehistoryfilewhentheshellisexited:
$history
[…]
25ps—forest
26history
27ps—forest
28history
$
$cat.bash_history
pwd
ls
history
exit
$
Noticewhenthe historycommandisrun,28commandsarelisted.Intheexample,the
listing is snipped for brevity. However, when the .bash_history file is displayed, only
fourcommandsarelisted,andtheydon’tmatchthehistorycommand’slist.
Youcanforcethecommandhistorytobewrittentothe.bash_historyfilebeforeleaving
ashellsession.Inordertoforcethiswrite,usethe-aoptiononthehistorycommand:
$history-a
$
$history
[…]
25ps—forest
26history
27ps—forest
28history
29ls-a
30cat.bash_history
31history-a
32history
$
$cat.bash_history
[…]
ps—forest
history
ps—forest
history
ls-a
cat.bash_history
history-a
Thistimebothlistingsneedtobesnippedbecausetheyaresolong.Noticethatcontents
fromboththe historycommandandthe .bash_historyfilematch,exceptforthevery
last command listed for the history command, because it came after the history -a
commandwasissued.
Note
Ifyouhavemultipleterminalsessionsopen,youcanstillappendthe.bash_history
ineachopensessionusingthehistory-acommand.However,thehistoriesarenot
automaticallyupdatedforyourotheropenterminalsessions.Thisisbecausethe
.bash_historyfileisreadonlywhenaterminalsessionisfirststarted.Toforcethe
.bash_historyfiletoberereadandaterminalsession’shistorytobeupdated,use
thehistory-ncommand.
Youcanrecallanycommandfromthehistorylist.Justenteranexclamationpointandthe
command’snumberfromthehistorylist:
$history
[…]
13pwd
14ls
15cd
16typepwd
17whichpwd
18typeecho
19whichecho
20type-apwd
21type-aecho
[…]
32history-a
33history
34cat.bash_history
35history
$
$!20
type-apwd
pwdisashellbuiltin
pwdis/bin/pwd
$
Commandnumber20waspulledfromcommandhistory.Noticethatsimilartoexecuting
thelastcommandinhistory,thebashshellfirstdisplaysthecommanditisrecallingfrom
theshell’shistory.Afterthecommandisdisplayed,itisexecuted.
Usingbashshellcommandhistorycanbeagreattimesaver.Youcandoevenmorewith
thebuilt-in history command. Be sure to view the bash manual pages for history,by
typingmanhistory.
Usingcommandaliases
The aliascommandisanothershellbuilt-incommand.Acommandaliasallowsyouto
create an alias name for common commands (along with their parameters) to help keep
yourtypingtoaminimum.
Mostlikely,yourLinuxdistributionhasalreadysetsomecommoncommandaliasesfor
you.Toseealistoftheactivealiases,usethealiascommandwiththe-pparameter:
$alias-p
[…]
aliasegrep=‘egrep—color=auto’
aliasfgrep=‘fgrep—color=auto’
aliasgrep=‘grep—color=auto’
aliasl=‘ls-CF’
aliasla=‘ls-A’
aliasll=‘ls-alF’
aliasls=‘ls—color=auto’
$
Noticethat,onthisUbuntuLinuxdistribution,analiasisusedtooverridethestandard ls
command. It automatically provides the —color parameter, indicating that the terminal
supportscolormodelistings.
Youcancreateyourownaliasesusingthealiascommand:
$aliasli=‘ls-li’
$
$li
total36
529581drwxr-xr-x.2ChristineChristine4096May1918:17Desktop
529585drwxr-xr-x.2ChristineChristine4096Apr2516:59Documents
529582drwxr-xr-x.2ChristineChristine4096Apr2516:59Downloads
529586drwxr-xr-x.2ChristineChristine4096Apr2516:59Music
529587drwxr-xr-x.2ChristineChristine4096Apr2516:59Pictures
529584drwxr-xr-x.2ChristineChristine4096Apr2516:59Public
529583drwxr-xr-x.2ChristineChristine4096Apr2516:59Templates
532891-rwxrw-r—.1ChristineChristine36May3007:21test.sh
529588drwxr-xr-x.2ChristineChristine4096Apr2516:59Videos
$
Afteryoudefineanaliasvalue,youcanuseitatanytimeinyourshell,includinginshell
scripts.Beawarethatbecausecommandaliasesarebuilt-incommands,analiasisvalid
onlyfortheshellprocessinwhichitisdefined:
$aliasli=‘ls-li’
$
$bash
$
$li
bash:li:commandnotfound
$
$exit
exit
$
Fortunately, you can make an alias value permanent across subshells. The next chapter
covershowtodothat,alongwithenvironmentvariables.
Summary
This chapter discussed the complicated interactive program, the GNU bash shell. It
coveredunderstandingtheshellprocessanditsrelationships,includinghowsubshellsare
spawnedandtheirrelationshiptotheparentshell.Wealsoexploredcommandsthatcreate
childprocessesandcommandsthatdon’t.
Thedefaultinteractiveshellisnormallystartedwheneverauserlogsintoaterminal.The
shell that the system starts depends upon a user ID configuration. Typically, it is
/bin/bash. The default system shell, /bin/sh, is used for system shell scripts, such as
thoseneededatstartup.
Asubshellorchildshellcanbespawnedusingthe bashcommand.Theyarealsocreated
whenaprocesslistorthe coproccommandisused.Usingsubshellsatthecommandline
canallowforcreativeandproductiveuseoftheCLI.Subshellscanbenested,spawning
grandchildshellsandgreat-grandchildshells.Creatingasubshellisanexpensiveprocess
asanewenvironmentfortheshellmustbecreatedaswell.
Finally,thechapterlookedattwodifferenttypesofshellcommands:built-inandexternal
commands. External commands create a child process with a new environment, but a
built-incommanddoesnot.Thiscausesexternalcommandstobemoreexpensivetouse.
Becauseanewenvironmentisnotneeded,built-incommandsaremoreefficientandnot
affectedbyanyenvironmentchanges.
Shells, subshells, processes, and forked processes are all affected by environment
variables. How the variables affect and can be used within these different contexts are
exploredinthenextchapter.
Chapter6
UsingLinuxEnvironmentVariables
InThisChapter
1. Lookingatenvironmentvariables
2. Creatingyourownlocalvariables
3. Removingvariables
4. Exploringdefaultshellenvironmentvariables
5. SettingthePATHenvironmentvariable
6. Locatingenvironmentfiles
7. Usingvariablearrays
LinuxenvironmentvariableshelpdefineyourLinuxshellexperience.Many
programsandscriptsuseenvironmentvariablestoobtainsysteminformationand
storetemporarydataandconfigurationinformation.Environmentvariablesaresetin
lotsofplacesontheLinuxsystem,andyoushouldknowwheretheseplacesare.
ThischapterwalksyouthroughtheworldofLinuxenvironmentvariables,showing
wheretheyare,howtousethem,andevenhowtocreateyourown.Thechapter
finishesoffwithhowtousevariablearrays.
ExploringEnvironmentVariables
Thebashshellusesafeaturecalledenvironmentvariablestostoreinformationaboutthe
shell session and the working environment (thus the name environment variables). This
feature also allows you to store data in memory that can be easily accessed by any
programorscriptrunningfromtheshell.Itisahandywaytostoreneededpersistentdata.
Therearetwoenvironmentvariabletypesinthebashshell:
Globalvariables
Localvariables
Thissectiondescribeseachtypeofenvironmentvariableandshowshowtoseeanduse
them.
Note
Eventhoughthebashshellusesspecificenvironmentvariablesthatareconsistent,
differentLinuxdistributionsoftenaddtheirownenvironmentvariables.The
environmentvariableexamplesyouseeinthischaptermaydifferslightlyfrom
what’savailableonyourspecificdistribution.Ifyourunintoanenvironmentvariable
notcoveredhere,checkyourLinuxdistribution’sdocumentation
Lookingatglobalenvironmentvariables
Global environment variables are visible from the shell session and from any spawned
child subshells. Local variables are available only in the shell that creates them. This
makes global environment variables useful in applications that create child subshells,
whichrequireparentshellinformation.
The Linux system sets several global environment variables when you start your bash
session.(Formoredetailsaboutwhatvariablesarestartedatthattime,seethe“Locating
System Environment Variables” section later in this chapter.) The system environment
variables almost always use all capital letters to differentiate them from normal user
environmentvariables.
Toviewglobalenvironmentvariables,usetheenvortheprintenvcommand:
$printenv
HOSTNAME=server01.class.edu
SELINUX_ROLE_REQUESTED=
TERM=xterm
SHELL=/bin/bash
HISTSIZE=1000
[…]
HOME=/home/Christine
LOGNAME=Christine
[…]
G_BROKEN_FILENAMES=1
_=/usr/bin/printenv
Somanyglobalenvironmentvariablesgetsetforthebashshellthatthedisplayhadtobe
snipped. Not only are many set during the login process, but how you log in can affect
whichonesaresetaswell.
To display an individual environment variable’s value, you can use the printenv
command,butnottheenvcommand:
$printenvHOME
/home/Christine
$
$envHOME
env:HOME:Nosuchfileordirectory
$
Youcanalsousethe echo command to display a variable’s value. When referencing an
environmentvariableinthiscase,youmustplaceadollarsign($)beforetheenvironment
variablename:
$echo$HOME
/home/Christine
$
Usingthedollarsignalongwiththevariablenamedoesmorethanjustdisplayitscurrent
definition when used with the echo command. The dollar sign before a variable name
allowsthevariabletobepassedasacommandparameter:
$ls$HOME
DesktopDownloadsMusicPublictest.sh
Documentsjunk.datPicturesTemplatesVideos
$
$ls/home/Christine
DesktopDownloadsMusicPublictest.sh
Documentsjunk.datPicturesTemplatesVideos
$
As mentioned earlier, global environment variables are also available to any process’s
subshells:
$bash
$
$ps-f
UIDPIDPPIDCSTIMETTYTIMECMD
50120172016016:00pts/000:00:00-bash
50120822017016:08pts/000:00:00bash
50120952082016:08pts/000:00:00ps-f
$
$echo$HOME
/home/Christine
$
$exit
exit
$
Inthisexample,afterspawningasubshellusingthebashcommand,theHOMEenvironment
variable’scurrentvalueisshown.Itissettotheexactsamevalue,/home/Christine,asit
wasintheparentshell.
Lookingatlocalenvironmentvariables
Localenvironmentvariables,astheirnameimplies,canbeseenonlyinthelocalprocess
inwhichtheyaredefined.Eventhoughtheyarelocal,theyarejustasimportantasglobal
environmentvariables.Infact,theLinuxsystemalsodefinesstandardlocalenvironment
variables for you by default. However, you can also define your own local variables.
These,asyouwouldassume,arecalleduser-definedlocalvariables.
TryingtoseethelocalvariableslistisalittletrickyattheCLI.Unfortunately,thereisn’ta
command that displays only these variables. The set command displays all variables
definedforaspecificprocess,includingbothlocalandglobalenvironmentvariablesand
user-definedvariables:
$set
BASH=/bin/bash
[…]
BASH_ALIASES=()
BASH_ARGC=()
BASH_ARGV=()
BASH_CMDS=()
BASH_LINENO=()
BASH_SOURCE=()
[…]
colors=/etc/DIR_COLORS
my_variable=‘HelloWorld’
[…]
$
Allglobalenvironmentvariablesdisplayedusingthe envor printenvcommandsappear
in the set command’s output. The additional environment variables are the local
environmentanduser-definedvariables.
Note
Thedifferencesbetweenthecommandsenv,printenv,andsetaresubtle.Theset
commanddisplaysbothglobalandlocalenvironmentvariablesanduser-defined
variables.Italsosortsthedisplayalphabetically.Theenvandprintenvaredifferent
fromsetinthattheydonotsortthevariables,nordotheyincludelocalenvironment
orlocaluser-definedvariables.Usedinthiscontext,envandprintenvproduce
duplicatelistings.However,theenvcommandhasadditionalfunctionalitythat
printenvdoesnothave,makingittheslightlymorepowerfulcommand.
SettingUser-DefinedVariables
Youcansetyourownvariablesdirectlyfromthebashshell.Thissectionshowsyouhow
tocreateyourownvariablesandreferencethemfromaninteractiveshellorshellscript
program.
Settinglocaluser-definedvariables
Afteryoustartabashshell(orspawnashellscript),you’reallowedtocreatelocaluserdefined variables that are visible within your shell process. You can assign either a
numericorastringvaluetoanenvironmentvariablebyassigningthevariabletoavalue
usingtheequalsign:
$echo$my_variable
$my_variable=Hello
$
$echo$my_variable
Hello
That was simple! Now, any time you need to reference the my_variable user-defined
variable’svalue,justreferenceitbythename$my_variable.
Ifyouneedtoassignastringvaluethatcontainsspaces,youneedtouseasingleordouble
quotationmarktodelineatethebeginningandtheendofthestring:
$my_variable=HelloWorld
-bash:World:commandnotfound
$
$my_variable=“HelloWorld”
$
$echo$my_variable
HelloWorld
$
Without the quotation marks, the bash shell assumes that the next word is another
commandtoprocess.Noticethatforthelocalvariableyoudefined,youusedlowercase
letters,whilethesystemenvironmentvariablesyou’veseensofarhavealluseduppercase
letters.
Tip
Thestandardbashshellconventionisforallenvironmentvariablestouseuppercase
letters.Ifyouarecreatingalocalvariableforyourselfandyourownshellscripts,use
lowercaseletters.Variablesarecasesensitive.Bykeepingyouruser-definedlocal
variableslowercase,youavoidthepotentialdisasterofredefiningasystem
environmentvariable.
It’s extremely important that you not use spaces between the variable name, the equal
sign,andthevalue.Ifyouputanyspacesintheassignment,thebashshellinterpretsthe
valueasaseparatecommand:
$my_variable=“HelloWorld”
-bash:my_variable:commandnotfound
$
After you set a local variable, it’s available for use anywhere within your shell process.
However,ifyouspawnanothershell,it’snotavailableinthechildshell:
$my_variable=“HelloWorld”
$
$bash
$
$echo$my_variable
$exit
exit
$
$echo$my_variable
HelloWorld
$
In this example, a child shell was spawned. The user-defined my_variable was not
availableinthechildshell.Thisisdemonstratedbytheblanklinereturnedafterthe echo
$my_variable command. After the child shell was exited and returned to the original
shell,thelocalvariablewasstillavailable.
Similarly,ifyousetalocalvariableinachildprocess,afteryouleavethechildprocess,
thelocalvariableisnolongeravailable:
$echo$my_child_variable
$bash
$
$my_child_variable=“HelloLittleWorld”
$
$echo$my_child_variable
HelloLittleWorld
$
$exit
exit
$
$echo$my_child_variable
$
Thelocalvariablesetwithinthechildshelldoesn’texistafterareturntotheparentshell.
You can change this behavior by turning your local user-defined variable into a global
environmentvariable.
Settingglobalenvironmentvariables
Global environment variables are visible from any child processes created by the parent
processthatsetsthevariable.Themethodusedtocreateaglobalenvironmentvariableis
tofirstcreatealocalvariableandthenexportittotheglobalenvironment.
Thisisdonebyusingtheexportcommandandthevariablenameminusthedollarsign:
$my_variable=“IamGlobalnow”
$
$exportmy_variable
$
$echo$my_variable
IamGlobalnow
$
$bash
$
$echo$my_variable
IamGlobalnow
$
$exit
exit
$
$echo$my_variable
IamGlobalnow
$
Afterdefiningandexportingthelocalvariable my_variable,achildshellwasstartedby
the bash command. The child shell was able to properly display the my_variable
variable’s value. The variable kept its value, because the export command made it a
globalenvironmentvariable.
Changingaglobalenvironmentvariablewithinachildshelldoesnotaffectthevariable’s
valueintheparentshell:
$my_variable=“IamGlobalnow”
$exportmy_variable
$
$echo$my_variable
IamGlobalnow
$
$bash
$
$echo$my_variable
IamGlobalnow
$
$my_variable=“Null”
$
$echo$my_variable
Null
$
$exit
exit
$
$echo$my_variable
IamGlobalnow
$
Afterdefiningandexportingthevariablemy_variable,asubshellwasstartedbythebash
command. The subshell properly displayed the value of the my_variable global
environmentvariable.Thevariable’svaluewasthenchangedbythechildshell.However,
thevariable’svaluewasmodifiedonlywithinthechildshellandnotintheparent’sshell.
A child shell cannot even use the export command to change the parent shell’s global
environmentvariable’svalue:
$my_variable=“IamGlobalnow”
$exportmy_variable
$
$echo$my_variable
IamGlobalnow
$
$bash
$
$echo$my_variable
IamGlobalnow
$
$my_variable=“Null”
$
$exportmy_variable
$
$echo$my_variable
Null
$
$exit
exit
$
$echo$my_variable
IamGlobalnow
$
Eventhoughthechildshellredefinedandexportedthevariable my_variable,theparent
shell’smy_variablevariablekeptitsoriginalvalue.
RemovingEnvironmentVariables
Ofcourse,ifyoucancreateanewenvironmentvariable,itmakessensethatyoucanalso
removeanexistingenvironmentvariable.Youcandothiswiththeunsetcommand.When
referencing the environment variable in the unset command, remember not to use the
dollarsign:
$echo$my_variable
IamGlobalnow
$
$unsetmy_variable
$
$echo$my_variable
$
Tip
Itcanbeconfusingtorememberwhentouseandwhennottousethedollarsignwith
environmentvariables.Justrememberthis:Ifyouaredoinganythingwiththe
variable,usethedollarsign.Ifyouaredoinganythingtothevariable,don’tusethe
dollarsign.Theexceptiontothisruleisusingprintenvtodisplayavariable’svalue.
Whendealingwithglobalenvironmentvariables,thingsgetalittletricky.Ifyou’reina
childprocessandunsetaglobalenvironmentvariable,itappliesonlytothechildprocess.
Theglobalenvironmentvariableisstillavailableintheparentprocess:
$my_variable=“IamGlobalnow”
$
$exportmy_variable
$
$echo$my_variable
IamGlobalnow
$
$bash
$
$echo$my_variable
IamGlobalnow
$
$unsetmy_variable
$
$echo$my_variable
$exit
exit
$
$echo$my_variable
IamGlobalnow
$
Justaswithmodifyingavariable,youcannotunsetitinachildshellandhavethevariable
beunsetintheparent’sshell.
UncoveringDefaultShellEnvironmentVariables
The bash shell uses specific environment variables by default to define the system
environment.Youcanalwayscountonthesevariablesbeingsetoravailabletobeseton
your Linux system. Because the bash shell is a derivative of the original Unix Bourne
shell,italsoincludesenvironmentvariablesoriginallydefinedinthatshell.
Table6.1showstheenvironmentvariablesthatthebashshellprovidesthatarecompatible
withtheoriginalUnixBourneshell.
Table6.1ThebashShellBourneVariables
Variable
CDPATH
HOME
IFS
MAIL
MAILPATH
OPTARG
OPTIND
PATH
PS1
PS2
Description
Acolon-separatedlistofdirectoriesusedasasearchpathforthecdcommand
Thecurrentuser’shomedirectory
Alistofcharactersthatseparatefieldsusedbytheshelltosplittextstrings
Thefilenameforthecurrentuser’smailbox(Thebashshellchecksthisfilefor
newmail.)
Acolon-separatedlistofmultiplefilenamesforthecurrentuser’smailbox
(Thebashshellcheckseachfileinthislistfornewmail.)
Thevalueofthelastoptionargumentprocessedbythegetoptcommand
Theindexvalueofthelastoptionargumentprocessedbythegetoptcommand
Acolon-separatedlistofdirectorieswheretheshelllooksforcommands
Theprimaryshellcommandlineinterfacepromptstring
Thesecondaryshellcommandlineinterfacepromptstring
Besides the default Bourne environment variables, the bash shell also provides a few
variablesofitsown,asshowninTable6.2.
Table6.2ThebashShellEnvironmentVariables
Variable
BASH
BASH_ALIASES
BASH_ARGC
BASH_ARCV
BASH_CMDS
BASH_COMMAND
Description
Thefullpathnametoexecutethecurrentinstanceofthebash
shell
Anassociativearrayofcurrentlysetaliases
Avariablearraythatcontainsthenumberofparametersbeing
passedtoasubroutineorshellscript
Avariablearraythatcontainstheparametersbeingpassedtoa
subroutineorshellscript
Anassociativearrayoflocationsofcommandstheshellhas
executed
Theshellcommandcurrentlybeingorabouttobeexecuted
Whenset,eachbashscriptattemptstoexecuteastartupfile
definedbythisvariablebeforerunning.
BASH_EXECUTION_STRING
Thecommand(s)passedusingthebash-coption
Avariablearraycontainingthesourcecodelinenumberofthe
BASH_LINENO
currentlyexecutingshellfunction
Aread-onlyvariablearraycontainingpatternsandtheirsubBASH_REMATCH
patternsforpositivematchesusingtheregularexpression
comparisonoperator,=∼
Avariablearraycontainingthesourcecodefilenameofthe
BASH_SOURCE
currentlyexecutingshellfunction
Thecurrentnestinglevelofasubshellenvironment(Theinitial
BASH_SUBSHELL
valueis0.)
Avariablearraythatcontainstheindividualmajorandminor
BASH_VERSINFO
versionnumbersofthecurrentinstanceofthebashshell
BASH_VERSION
Theversionnumberofthecurrentinstanceofthebashshell
Ifsettoavalidfiledescriptor(0,1,2),traceoutputgenerated
BASH_XTRACEFD
fromthe‘set-x’debuggingoptioncanberedirected.Thisis
oftenusedtoseparatetraceoutputintoafile.
BASHOPTS
Alistofbashshelloptionsthatarecurrentlyenabled
BASHPID
ProcessIDofthecurrentbashprocess
Containstheterminalwidthoftheterminalusedforthecurrent
COLUMNS
instanceofthebashshell
AnindexintothevariableCOMP_WORDS,whichcontainsthe
COMP_CWORD
currentcursorposition
COMP_LINE
Thecurrentcommandline
Theindexofthecurrentcursorpositionrelativetothe
COMP_POINT
beginningofthecurrentcommand
Thefinalkeyusedtoinvokethecurrentcompletionofashell
COMP_KEY
function
Anintegervaluerepresentingthetypeofcompletionattempted
COMP_TYPE
thatcausedacompletionshellfunctiontobeinvoked
TheReadlinelibrarywordseparatorcharactersforperforming
COMP_WORDBREAKS
wordcompletion
Anarrayvariablethatcontainstheindividualwordsonthe
COMP_WORDS
currentcommandline
Anarrayvariablethatcontainsthepossiblecompletioncodes
COMPREPLY
generatedbyashellfunction
BASH_ENV
Variable
COPROC
Description
Anarrayvariablethatholdsanunnamedcoprocess’I/Ofile
descriptors
DIRSTACK
EMACS
ENV
EUID
FCEDIT
FIGNORE
FUNCNAME
FUNCNEST
GLOBIGNORE
GROUPS
histchars
HISTCMD
HISTCONTROL
HISTFILE
HISTFILESIZE
HISTTIMEFORMAT
HISTIGNORE
HISTSIZE
HOSTFILE
HOSTNAME
HOSTTYPE
IGNOREEOF
INPUTRC
LANG
Anarrayvariablethatcontainsthecurrentcontentsofthe
directorystack
Indicatestheemacsshellbufferisexecutingandlineeditingis
disabled,whensetto‘t’
Whenset,executesthestartupfiledefinedbeforeabashshell
scriptruns(Itisusedonlywhenthebashshellhasbeen
invokedinPOSIXmode.)
ThenumericeffectiveuserIDofthecurrentuser
Thedefaulteditorusedbythefccommand
Acolon-separatedlistofsuffixestoignorewhenperforming
filenamecompletion
Thenameofthecurrentlyexecutingshellfunction
Setsthemaximumallowedfunctionnestinglevel,whensetto
anumbergreaterthanzero(Ifitisexceeded,thecurrent
commandaborts.)
Acolon-separatedlistofpatternsdefiningthesetoffilenames
tobeignoredbyfilenameexpansion
Avariablearraycontainingthelistofgroupsofwhichthe
currentuserisamember
Uptothreecharacters,whichcontrolhistoryexpansion
Thehistorynumberofthecurrentcommand
Controlswhatcommandsareenteredintheshellhistorylist
Thenameofthefileinwhichtosavetheshellhistorylist
(.bash_historybydefault)
Themaximumnumberoflinestosaveinthehistoryfile
Usedasaformattingstringtoprinteachcommand’stimestamp
inbashhistory,ifsetandnotnull
Acolon-separatedlistofpatternsusedtodecidewhich
commandsareignoredforthehistoryfile
Themaximumnumberofcommandsstoredinthehistoryfile
Containsthenameofthefilethatshouldbereadwhentheshell
needstocompleteahostname
Thenameofthecurrenthost
Astringdescribingthemachinethebashshellisrunningon
ThenumberofconsecutiveEOFcharacterstheshellmust
receivebeforeexiting(Ifthisvaluedoesn’texist,thedefaultis
1.)
ThenameoftheReadlineinitializationfile(Thedefaultis
.inputrc.)
Thelocalecategoryfortheshell
LC_ALL
LC_COLLATE
LC_CTYPE
LC_MESSAGES
LC_NUMERIC
LINENO
LINES
MACHTYPE
MAPFILE
MAILCHECK
OLDPWD
OPTERR
OSTYPE
PIPESTATUS
POSIXLY_CORRECT
PPID
PROMPT_COMMAND
PROMPT_DIRTRIM
PS3
PS4
PWD
Variable
RANDOM
READLINE_LINE
READLINE_POINT
OverridestheLANGvariable,definingalocalecategory
Setsthecollationorderusedwhensortingstringvalues
Determinestheinterpretationofcharactersusedinfilename
expansionandpatternmatching
Determinesthelocalesettingusedwheninterpretingdoublequotedstringsprecededbyadollarsign
Determinesthelocalesettingusedwhenformattingnumbers
Thelinenumberinascriptcurrentlyexecuting
Definesthenumberoflinesavailableontheterminal
Astringdefiningthesystemtypeincpu-company-system
format
Anarrayvariablethatholdsread-intextfromthemapfile
commandwhennoarrayvariablenameisgiven
Howoften(inseconds)theshellshouldcheckfornewmail
(Thedefaultis60.)
Thepreviousworkingdirectoryusedintheshell
Ifsetto1,thebashshelldisplayserrorsgeneratedbythe
getoptscommand.
Astringdefiningtheoperatingsystemtheshellisrunningon
Avariablearraycontainingalistofexitstatusvaluesfromthe
processesintheforegroundprocess
Ifset,bashstartsinPOSIXmode.
TheprocessID(PID)ofthebashshell’sparentprocess
Ifset,thecommandtoexecutebeforedisplayingtheprimary
prompt
Anintegerusedtoindicatethenumberoftrailingdirectory
namestodisplaywhenusingthe∖wand∖Wpromptstring
escapes(Thedirectorynamesremovedarereplacedwithone
setofellipses.)
Theprompttousefortheselectcommand
Thepromptdisplayedbeforethecommandlineisechoedifthe
bash-xparameterisused
Thecurrentworkingdirectory
Description
Returnsarandomnumberbetween0and32767(Assigninga
valuetothisvariableseedsthepseudo-randomnumber
generator.)
Readlinebuffercontentswhenusingbind-xcommand
Readlinebuffercontentinsertionpoint’scurrentpositionwhen
REPLY
SECONDS
SHELL
SHELLOPTS
SHLVL
TIMEFORMAT
TMOUT
TMPDIR
UID
usingbind-xcommand
Thedefaultvariableforthereadcommand
Thenumberofsecondssincetheshellwasstarted(Assigninga
valueresetsthetimertothevalue.)
Thefullpathnametothebashshell
Acolon-separatedlistofenabledbashshelloptions
Indicatestheshelllevel,incrementedbyoneeachtimeanew
bashshellisstarted
Aformatspecifyinghowtheshelldisplaystimevalues
Thevalueofhowlong(inseconds)theselectandread
commandsshouldwaitforinput(Thedefaultofzeroindicates
towaitindefinitely.)
Directorynamewherethebashshellcreatestemporaryfilesfor
itsuse
ThenumericrealuserIDofthecurrentuser
You may notice that not all default environment variables are shown when the set
commandisused.Whennotinuse,thedefaultenvironmentvariablesarenotallrequired
tocontainavalue.
SettingthePATHEnvironmentVariable
Whenyouenteranexternalcommand(seeChapter5)intheshellcommandlineinterface
(CLI), the shell must search the system to find the program. The PATH environment
variabledefinesthedirectoriesitsearcheslookingforcommandsandprograms.Onthis
UbuntuLinuxsystem,thePATHenvironmentvariablelookslikethis:
$echo$PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:
/sbin:/bin:/usr/games:/usr/local/games
$
This shows that there are eight directories where the shell looks for commands and
programs.ThedirectoriesinthePATHareseparatedbycolons.
Ifacommand’sorprogram’slocationisnotincludedinthePATHvariable,theshellcannot
find it without an absolute directory reference. If the shell cannot find the command or
program,itproducesanerrormessage:
$myprog
-bash:myprog:commandnotfound
$
Theproblemisthatoftenapplicationsplacetheirexecutableprogramsindirectoriesthat
aren’tinthe PATHenvironmentvariable.Thetrickisensuringthatyour PATHenvironment
variableincludesallthedirectorieswhereyourapplicationsreside.
You can add new search directories to the existing PATH environment variable without
havingtorebuilditfromscratch.TheindividualdirectorieslistedinthePATHareseparated
by colons. All you need to do is reference the original PATH value and add any new
directoriestothestring.Thislookssomethinglikethis:
$echo$PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:
/sbin:/bin:/usr/games:/usr/local/games
$
$PATH=$PATH:/home/christine/Scripts
$
$echo$PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/
games:/usr/local/games:/home/christine/Scripts
$
$myprog
Thefactorialof5is120.
$
By adding the directory to the PATH environment variable, you can now execute your
programfromanywhereinthevirtualdirectorystructure:
$cd/etc
$
$myprog
Thefactorialof5is120
$
Tip
Ifyouwantyourprogram’slocationtobeavailabletosubshells,besuretoexport
yourmodifiedPATHenvironmentvariable.
A common trick for programmers is to include the single dot symbol in their PATH
environmentvariable.Thesingledotsymbolrepresentsthecurrentdirectory(seeChapter
3):
$PATH=$PATH:.
$
$cd/home/christine/Old_Scripts
$
$myprog2
Thefactorialof6is720
$
ChangestothePATHvariablelastonlyuntilyouexitthesystemorthesystemreboots.
Thechangesarenotpersistent.Inthenextsection,youseehowyoucanmakechangesto
environmentvariablespermanent.
LocatingSystemEnvironmentVariables
TheLinuxsystemusesenvironmentvariablesformanypurposes.Youknownowhowto
modifysystemenvironmentvariablesandcreateyourownvariables.Thetrickisinhow
theseenvironmentvariablesaremadepersistent.
When you start a bash shell by logging in to the Linux system, by default bash checks
severalfilesforcommands.Thesefilesarecalledstartupfilesorenvironmentfiles.The
startupfilesthatbashprocessesdependonthemethodyouusetostartthebashshell.You
canstartabashshellinthreeways:
Asadefaultloginshellatlogintime
Asaninteractiveshellthatisstartedbyspawningasubshell
Asanon-interactiveshelltorunascript
Thefollowingsectionsdescribethestartupfilesthebashshellexecutesineachofthese
startupmethods.
Understandingtheloginshellprocess
WhenyoulogintotheLinuxsystem,thebashshellstartsasaloginshell.Theloginshell
typicallylooksforfivedifferentstartupfilestoprocesscommandsfrom:
/etc/profile
$HOME/.bash_profile
$HOME/.bashrc
$HOME/.bash_login
$HOME/.profile
The/etc/profilefileisthemaindefaultstartupfileforthebashshellonthesystem.All
usersonthesystemexecutethisstartupfilewhentheylogin.
Note
BeawarethatsomeLinuxdistributionsusePluggableAuthenticationModules
(PAM).Inthiscase,beforethebashshellisstarted,PAMfilesareprocessed,
includingonesthatmaycontainenvironmentvariables.PAMfileexamplesinclude
the/etc/environmentfileandthe$HOME/.pam_environmentfile.Findmore
informationaboutPAMathttp://linux-pam.org.
The other four startup files are specific for each user and can be customized for an
individualuser’srequirements.Let’slookcloseratthesefiles.
Viewingthe/etc/profilefile
The/etc/profilefileisthemaindefaultstartupfileforthebashshell.Wheneveryoulog
intotheLinuxsystem,bashexecutesthecommandsinthe/etc/profilestartupfilefirst.
DifferentLinuxdistributionsplacedifferentcommandsinthisfile.OnthisUbuntuLinux
system,thefilelookslikethis:
$cat/etc/profile
#/etc/profile:system-wide.profilefilefortheBourneshell(sh(1))
#andBournecompatibleshells(bash(1),ksh(1),ash(1),…).
if[”$PS1”];then
if[”$BASH”]&&[”$BASH”!=”/bin/sh”];then
#Thefilebash.bashrcalreadysetsthedefaultPS1.
#PS1=’\h:\w\$‘
if[-f/etc/bash.bashrc];then
./etc/bash.bashrc
fi
else
if[”`id-u`”-eq0];then
PS1=’#‘
else
PS1=’$‘
fi
fi
fi
#Thedefaultumaskisnowhandledbypam_umask.
#Seepam_umask(8)and/etc/login.defs.
if[-d/etc/profile.d];then
foriin/etc/profile.d/*.sh;do
if[-r$i];then
.$i
fi
done
unseti
fi
$
Most of the commands and syntax you see in this file are covered in more detail in
Chapter12andbeyond.Eachdistribution’s /etc/profile file has different settings and
commands. For example, notice that a file is mentioned in this Ubuntu distribution’s
/etc/profile file above, called /etc/bash.bashrc. It contains system environment
variables.
However, in this CentOS distribution’s /etc/profile file listed below, no
/etc/bash.bashrc file is called. Also note that it sets and exports some system
environmentvariableswithinitself:
$cat/etc/profile
#/etc/profile
#Systemwideenvironmentandstartupprograms,forloginsetup
#Functionsandaliasesgoin/etc/bashrc
#It’sNOTagoodideatochangethisfileunlessyouknowwhatyou
#aredoing.It’smuchbettertocreateacustom.shshellscriptin
#/etc/profile.d/tomakecustomchangestoyourenvironment,to
#preventtheneedformerginginfutureupdates.
pathmunge(){
case“:${PATH}:”in
*:”$1”:*)
;;
*)
if[“$2”=“after”];then
PATH=$PATH:$1
else
PATH=$1:$PATH
fi
esac
}
if[-x/usr/bin/id];then
if[-z“$EUID”];then
#kshworkaround
EUID=`id-u`
UID=`id-ru`
fi
USER=”`id-un`”
LOGNAME=$USER
MAIL=”/var/spool/mail/$USER”
fi
#Pathmanipulation
if[“$EUID”=“0”];then
pathmunge/sbin
pathmunge/usr/sbin
pathmunge/usr/local/sbin
else
pathmunge/usr/local/sbinafter
pathmunge/usr/sbinafter
pathmunge/sbinafter
fi
HOSTNAME=`/bin/hostname2>/dev/null`
HISTSIZE=1000
if[“$HISTCONTROL”=“ignorespace”];then
exportHISTCONTROL=ignoreboth
else
exportHISTCONTROL=ignoredups
fi
exportPATHUSERLOGNAMEMAILHOSTNAMEHISTSIZEHISTCONTROL
#Bydefault,wewantumasktogetset.Thissetsitforloginshell
#Currentthresholdforsystemreserveduid/gidsis200
#Youcouldcheckuidgidreservationvalidityin
#/usr/share/doc/setup-*/uidgidfile
if[$UID-gt199]&&[“`id-gn`”=“`id-un`”];then
umask002
else
umask022
fi
foriin/etc/profile.d/*.sh;do
if[-r“$i”];then
if[“${-#*i}”!=“$-”];then
.“$i”
else
.“$i”>/dev/null2>&1
fi
fi
done
unseti
unset-fpathmunge
$
Both distributions’ /etc/profile files use a certain feature. It is a for statement that
iterates through any files located in the /etc/profile.d directory. (for statements are
discussed in detail in Chapter 13.) This provides a place for the Linux system to place
application-specific startup files that is executed by the shell when you log in. On this
UbuntuLinuxsystem,thefollowingfilesareintheprofile.ddirectory:
$ls-l/etc/profile.d
total12
-rw-r—r—1rootroot40Apr1506:26appmenu-qt5.sh
-rw-r—r—1rootroot663Apr710:10bash_completion.sh
-rw-r—r—1rootroot1947Nov222013vte.sh
$
YoucanseethatthisCentOssystemhasquiteafewmorefilesin/etc/profile.d:
$ls-l/etc/profile.d
total80
-rw-r—r—.1rootroot1127Mar507:17colorls.csh
-rw-r—r—.1rootroot1143Mar507:17colorls.sh
-rw-r—r—.1rootroot92Nov222013cvs.csh
-rw-r—r—.1rootroot78Nov222013cvs.sh
-rw-r—r—.1rootroot192Feb2409:24glib2.csh
-rw-r—r—.1rootroot192Feb2409:24glib2.sh
-rw-r—r—.1rootroot58Nov222013gnome-ssh-askpass.csh
-rw-r—r—.1rootroot70Nov222013gnome-ssh-askpass.sh
-rwxr-xr-x.1rootroot373Sep232009kde.csh
-rwxr-xr-x.1rootroot288Sep232009kde.sh
-rw-r—r—.1rootroot1741Feb2005:44lang.csh
-rw-r—r—.1rootroot2706Feb2005:44lang.sh
-rw-r—r—.1rootroot122Feb72007less.csh
-rw-r—r—.1rootroot108Feb72007less.sh
-rw-r—r—.1rootroot976Sep232011qt.csh
-rw-r—r—.1rootroot912Sep232011qt.sh
-rw-r—r—.1rootroot2142Mar1315:37udisks-bash-completion.sh
-rw-r—r—.1rootroot97Apr52012vim.csh
-rw-r—r—.1rootroot269Apr52012vim.sh
-rw-r—r—.1rootroot169May202009which2.sh
$
Notice that several files are related to specific applications on the system. Most
applicationscreatetwostartupfiles—oneforthebashshell(usingthe.shextension)and
oneforthecshell(usingthe.cshextension).
The lang.cshand lang.sh files attempt to determine the default language character set
usedonthesystemandsettheLANGenvironmentvariableappropriately.
Viewingthe$HOMEstartupfiles
Theremainingstartupfilesareallusedforthesamefunction—toprovideauser-specific
startupfilefordefininguser-specificenvironmentvariables.MostLinuxdistributionsuse
onlyoneortwoofthesefourstartupfiles:
$HOME/.bash_profile
$HOME/.bashrc
$HOME/.bash_login
$HOME/.profile
Noticethatallfourfilesstartwithadot,makingthemhiddenfiles(theydon’tappearina
normal lscommandlisting).Becausetheyareintheuser’s HOMEdirectory,eachusercan
editthefilesandaddhisorherownenvironmentvariablesthatareactiveforeverybash
shellsessiontheystart.
Note
EnvironmentfilesareoneareawhereLinuxdistributionsvarygreatly.Notevery
$HOMEfilelistedinthissectionexistsforeveryuser.Forexample,someusersmay
haveonlythe$HOME/.bash_profilefile.Thisisnormal.
Thefirstfilefoundinthefollowingorderedlistisrun,andtherestareignored:
$HOME/.bash_profile
$HOME/.bash_login
$HOME/.profile
Noticethat$HOME/.bashrcisnotinthislist.Thisisbecauseitistypicallyrunfromoneof
theotherfiles.
Tip
Rememberthat$HOMErepresentsauser’shomedirectory.Also,thetilde(∼)isusedto
representauser’shomedirectory.
ThisCentOSLinuxsystemcontainsthefollowing.bash_profilefile:
$cat$HOME/.bash_profile
#.bash_profile
#Getthealiasesandfunctions
if[-f~/.bashrc];then
.~/.bashrc
fi
#Userspecificenvironmentandstartupprograms
PATH=$PATH:$HOME/bin
exportPATH
$
The.bash_profilestartupfilefirstcheckstoseeifthestartupfile,.bashrc,ispresentin
theHOMEdirectory.Ifit’sthere,thestartupfileexecutesthecommandsinit.
Understandingtheinteractiveshellprocess
If you start a bash shell without logging into a system (if you just type bash at a CLI
prompt, for example), you start what’s called an interactive shell. The interactive shell
doesn’t act like the login shell, but it still provides a CLI prompt for you to enter
commands.
Ifbashisstartedasaninteractiveshell,itdoesn’tprocessthe /etc/profilefile.Instead,
itonlychecksforthe.bashrcfileintheuser’sHOMEdirectory.
OnthisLinuxCentOSdistribution,thisfilelookslikethis:
$cat.bashrc
#.bashrc
#Sourceglobaldefinitions
if[-f/etc/bashrc];then
./etc/bashrc
fi
#Userspecificaliasesandfunctions
$
The .bashrcfiledoestwothings.First,itchecksforacommon bashrcfileinthe /etc
directory. Second, it provides a place for the user to enter personal command aliases
(discussedinChapter5)andprivatescriptfunctions(describedinChapter17).
Understandingthenon-interactiveshellprocess
Thelasttypeofshellisanon-interactivesubshell.Thisistheshellwherethesystemcan
starttoexecuteashellscript.Thisisdifferentinthatthereisn’taCLIprompttoworry
about. However, you may want to run specific startup commands each time you start a
scriptonyoursystem.
Tip
Scriptscanbeexecutedindifferentways.Onlysomeexecutionmethodsstarta
subshell.YoulearnaboutthedifferentshellexecutionmethodsinChapter11.
To accommodate that situation, the bash shell provides the BASH_ENV environment
variable. When the shell starts a non-interactive subshell process, it checks this
environment variable for the startup file name to execute. If one is present, the shell
executesthefile’scommands,whichtypicallyincludevariablessetfortheshellscripts.
OnthisCentOSLinuxdistribution,thisenvironmentvalueisnotsetbydefault.Whena
variableisnotset,theprintenvcommandsimplyreturnstheCLIprompt:
$printenvBASH_ENV
$
OnthisUbuntudistribution,theBASH_ENVvariableisn’tseteither.Rememberthat,whena
variableisnotset,theechocommanddisplaysablanklineandreturnstheCLIprompt:
$echo$BASH_ENV
$
So if the BASH_ENV variable isn’t set, how do the shell scripts get their environment
variables?Rememberthatsomeshellscriptexecutionmethodsstartasubshell,alsocalled
achildshell(seeChapter5).Achildshellinheritsitsparentshell’sexportedvariables.
Forexample,iftheparentshellwasaloginshellandhadvariablessetandexportedinthe
/etc/profile file, /etc/profile.d/*.sh files, and the $HOME/.bashrc file, the child
shellforthescriptinheritsthesevariables.
However,rememberthatanyvariablessetbutnotexportedbytheparentshellarelocal
variables.Localvariablesarenotinheritedbyasubshell.
For scripts that do not start a subshell, the variables are already available in the current
shell.Thus,evenif BASH_ENVisnotset,boththecurrentshell’slocalandglobalvariables
arepresenttobeused.
Makingenvironmentvariablespersistent
Now that you know you way around the various shell process types and their various
environmentfiles,locatingthepermanentenvironmentvariablesismucheasier.Youcan
alsosetyourownpermanentglobalorlocalvariablesusingthesefiles.
For global environment variables (those variables needed by all the users on a Linux
system),itmaybetemptingtoputnewormodifiedvariablesettingsinthe/etc/profile,
butthisisabadidea.Thefilecouldbechangedwhenyourdistributionisupgraded,and
youwouldloseallthecustomizedvariablesettings.
Itisabetterideatocreateafileendingwith .shinthe /etc/profile.ddirectory.Inthat
file,placeallyournewormodifiedglobalenvironmentvariablesettings.
On most distributions, the best place to store an individual user’s persistent bash shell
variablesisinthe $HOME/.bashrcfile.Thisistrueforallshellprocesstypes.However,if
theBASH_ENVvariableisset,keepinmindthatunlessitpointsto$HOME/.bashrc,youmay
needtostoreauser’svariablesfornon-interactiveshelltypeselsewhere.
Note
Keepinmindthatuserenvironmentvariablesforgraphicalinterfaceelements,such
astheGUIclient,mayneedtobesetindifferentconfigurationfilesthanwherebash
shellenvironmentvariablesareset
RecallbackinChapter5thatcommandaliassettingsarealsonotpersistent.Youcanalso
store your personal alias settings in the $HOME/.bashrc startup file to make them
permanent.
LearningaboutVariableArrays
Areallycoolfeatureofenvironmentvariablesisthattheycanbeusedasarrays.Anarray
isavariablethatcanholdmultiplevalues.Valuescanbereferencedeitherindividuallyor
asawholefortheentirearray.
To set multiple values for an environment variable, just list them in parentheses, with
valuesseparatedbyspaces:
$mytest=(onetwothreefourfive)
$
Not much excitement there. If you try to display the array as a normal environment
variable,you’llbedisappointed:
$echo$mytest
one
$
Only the first value in the array appears. To reference an individual array element, you
must use a numerical index value, which represents its place in the array. The numeric
valueisenclosedinsquarebrackets:
$echo${mytest[2]}
three
$
Tip
Environmentvariablearraysstartwithanindexvalueofzero.Thiscanbeconfusing.
To display an entire array variable, you use the asterisk wildcard character as the index
value:
$echo${mytest[*]}
onetwothreefourfive
$
Youcanalsochangethevalueofanindividualindexposition:
$mytest[2]=seven
$
$echo${mytest[*]}
onetwosevenfourfive
$
Youcanevenusethe unsetcommandtoremoveanindividualvaluewithinthearray,but
becareful,becausethisgetstricky.Watchthisexample:
$unsetmytest[2]
$
$echo${mytest[*]}
onetwofourfive
$
$echo${mytest[2]}
$echo${mytest[3]}
four
$
Thisexampleusesthe unsetcommandtoremovethevalueatindexvalue2.Whenyou
displaythearray,itappearsthattheotherindexvaluesjustdroppeddownone.However,
ifyouspecificallydisplaythedataatindexvalue2,youseethatthatlocationisempty.
Finally, you can remove the entire array just by using the array name in the unset
command:
$unsetmytest
$
$echo${mytest[*]}
$
Sometimesvariablearraysjustcomplicatematters,sothey’reoftennotusedinshellscript
programming.They’renotveryportabletoothershellenvironments,whichisadownside
ifyoudolotsofshellprogrammingfordifferentshells.Somebashsystemenvironment
variables use arrays (such as BASH_VERSINFO), but overall you probably won’t run into
themveryoften.
Summary
This chapter examined the world of Linux environment variables. Global environment
variables can be accessed from any child shell spawned by the parent shell in which
they’re defined. Local environment variables can be accessed only from the process in
whichthey’redefined.
TheLinuxsystemusesbothglobalandlocalenvironmentvariablestostoreinformation
aboutthesystemenvironment.Youcanaccessthisinformationfromtheshellcommand
lineinterface,aswellaswithinshellscripts.Thebashshellusesthesystemenvironment
variables defined in the original Unix Bourne shell, as well as lots of new environment
variables.ThePATHenvironmentvariabledefinesthesearchpatternthebashshelltakesto
findanexecutablecommand.Youcanmodifythe PATHenvironmentvariabletoaddyour
own directories, or even the current directory symbol, to make running your programs
easier.
Youcanalsocreateyourownglobalandlocalenvironmentvariablesforyourownuse.
After you create an environment variable, it’s accessible for the entire duration of your
shellsession.
The bash shell executes several startup files when it starts up. These startup files can
contain environment variable definitions to set standard environment variables for each
bash session. When you log in to the Linux system, the bash shell accesses the
/etc/profile startup file and three local startup files for each user,
$HOME/.bash_profile, $HOME/.bash_login, and $HOME/.profile. Users can customize
thesefilestoincludeenvironmentvariablesandstartupscriptsfortheirownuse.
Finally,thischapterdiscussedtheuseofenvironmentvariablearrays.Theseenvironment
variablescancontainmultiplevaluesinasinglevariable.Youcanaccessthevalueseither
individually by referencing an index value or as a whole by referencing the entire
environmentvariablearrayname.
ThenextchapterdivesintotheworldofLinuxfilepermissions.Thisispossiblythemost
difficult topic for novice Linux users. However, to write good shell scripts, you need to
understandhowfilepermissionsworkandbeabletousethemonyourLinuxsystem.
Chapter7
UnderstandingLinuxFilePermissions
InThisChapter
1. UnderstandingLinuxsecurity
2. Decodingthepermissions
3. WorkingwithLinuxgroups
Nosystemiscompletewithoutsomeformofsecurity.Theremustbeamechanism
availabletoprotectfilesfromunauthorizedviewingormodification.TheLinux
systemfollowstheUnixmethodoffilepermissions,allowingindividualusersand
groupsaccesstofilesbasedonasetofsecuritysettingsforeachfileanddirectory.
ThischapterdiscusseshowtousetheLinuxfilesecuritysystemtoprotectdatawhen
necessaryandsharedatawhendesired.
LinuxSecurity
ThecoreoftheLinuxsecuritysystemistheuseraccount.Eachindividualwhoaccessesa
Linux system should have a unique user account assigned. The users’ permissions to
objectsonthesystemdependontheuseraccounttheyloginwith.
UserpermissionsaretrackedusingauserID(oftencalledaUID),whichisassignedtoan
accountwhenit’screated.TheUIDisanumericalvalue,uniqueforeachuser.However,
youdon’tlogintoaLinuxsystemusingyourUID.Instead,youusealoginname.The
loginnameisanalphanumerictextstringofeightcharactersorfewerthattheuserusesto
logintothesystem(alongwithanassociatedpassword).
TheLinuxsystemusesspecialfilesandutilitiestotrackandmanageuseraccountsonthe
system. Before we can discuss file permissions, we need to discuss how Linux handles
user accounts. This section describes the files and utilities required for user accounts so
thatyoucanunderstandhowtousethemwhenworkingwithfilepermissions.
The/etc/passwdfile
The Linux system uses a special file to match the login name to a corresponding UID
value.Thisfileisthe /etc/passwdfile.The /etc/passwdfilecontainsseveralpiecesof
informationabouttheuser.Here’swhatatypical /etc/passwdfilelookslikeonaLinux
system:
$cat/etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
news:x:9:13:news:/etc/news:
uucp:x:10:14:uucp:/var/spool/uucp:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
gopher:x:13:30:gopher:/var/gopher:/sbin/nologin
ftp:x:14:50:FTPUser:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
rpm:x:37:37::/var/lib/rpm:/sbin/nologin
vcsa:x:69:69:virtualconsolememoryowner:/dev:/sbin/nologin
mailnull:x:47:47::/var/spool/mqueue:/sbin/nologin
smmsp:x:51:51::/var/spool/mqueue:/sbin/nologin
apache:x:48:48:Apache:/var/www:/sbin/nologin
rpc:x:32:32:RpcbindDaemon:/var/lib/rpcbind:/sbin/nologin
ntp:x:38:38::/etc/ntp:/sbin/nologin
nscd:x:28:28:NSCDDaemon:/:/sbin/nologin
tcpdump:x:72:72::/:/sbin/nologin
dbus:x:81:81:Systemmessagebus:/:/sbin/nologin
avahi:x:70:70:Avahidaemon:/:/sbin/nologin
hsqldb:x:96:96::/var/lib/hsqldb:/sbin/nologin
sshd:x:74:74:Privilege-separatedSSH:/var/empty/sshd:/sbin/nologin
rpcuser:x:29:29:RPCServiceUser:/var/lib/nfs:/sbin/nologin
nfsnobody:x:65534:65534:AnonymousNFSUser:/var/lib/nfs:/sbin/nologin
haldaemon:x:68:68:HALdaemon:/:/sbin/nologin
xfs:x:43:43:XFontServer:/etc/X11/fs:/sbin/nologin
gdm:x:42:42::/var/gdm:/sbin/nologin
rich:x:500:500:RichBlum:/home/rich:/bin/bash
mama:x:501:501:Mama:/home/mama:/bin/bash
katie:x:502:502:katie:/home/katie:/bin/bash
jessica:x:503:503:Jessica:/home/jessica:/bin/bash
mysql:x:27:27:MySQLServer:/var/lib/mysql:/bin/bash
$
The root user account is the administrator for the Linux system and is always assigned
UID 0. As you can see, the Linux system creates lots of user accounts for various
functionsthataren’tactualusers.Thesearecalledsystemaccounts.Asystemaccountisa
specialaccountthatservicesrunningonthesystemusetogainaccesstoresourcesonthe
system.AllservicesthatruninbackgroundmodeneedtobeloggedintotheLinuxsystem
underasystemuseraccount.
Beforesecuritybecameabigissue,theseservicesoftenjustloggedinusingthe rootuser
account. Unfortunately, if an unauthorized person broke into one of these services, he
instantly gained access to the system as the root user. To prevent this, now just about
everyservicethatrunsinbackgroundonaLinuxserverhasitsownuseraccounttologin
with. This way, if a troublemaker compromises a service, he still can’t necessarily get
accesstothewholesystem.
LinuxreservesUIDsbelow500forsystemaccounts.Someservicesevenrequirespecific
UIDstoworkproperly.Whenyoucreateaccountsfornormalusers,mostLinuxsystems
assignthefirstavailableUIDstartingat500(althoughthisisnotnecessarilytrueforall
Linuxdistributions).
You probably noticed that the /etc/passwd file contains much more than just the login
name and UID for the user. The fields of the /etc/passwd file contain the following
information:
Theloginusername
Thepasswordfortheuser
ThenumericalUIDoftheuseraccount
ThenumericalgroupID(GID)oftheuseraccount
Atextdescriptionoftheuseraccount(calledthecommentfield)
ThelocationoftheHOMEdirectoryfortheuser
Thedefaultshellfortheuser
Thepasswordfieldinthe /etc/passwdfileissettoan x.Thisdoesn’tmeanthatallthe
user accounts have the same password. In the old days of Linux, the /etc/passwd file
containedanencryptedversionoftheuser’spassword.However,becauselotsofprograms
needtoaccessthe /etc/passwdfileforuserinformation,thisbecameasecurityproblem.
Withtheadventofsoftwarethatcouldeasilydecryptencryptedpasswords,thebadguys
had a field day trying to break user passwords stored in the /etc/passwd file. Linux
developersneededtorethinkthatpolicy.
Now,mostLinuxsystemsholduserpasswordsinaseparatefile(calledtheshadowfile,
locatedat /etc/shadow).Onlyspecialprograms(suchastheloginprogram)areallowed
accesstothisfile.
The /etc/passwd file is a standard text file. You can use any text editor to manually
perform user management functions (such as adding, modifying, or removing user
accounts) directly in the /etc/passwd file. However, this is an extremely dangerous
practice.Ifthe/etc/passwdfilebecomescorrupt,thesystemcan’treadit,anditprevents
anyone(eventherootuser)fromloggingin.Instead,it’ssafertousethestandardLinux
usermanagementutilitiestoperformallusermanagementfunctions.
The/etc/shadowfile
The /etc/shadow file provides more control over how the Linux system manages
passwords.Onlytherootuserhasaccesstothe /etc/shadowfile,makingitmoresecure
thanthe/etc/passwdfile.
The /etc/shadowfilecontainsonerecordforeachuseraccountonthesystem.Arecord
lookslikethis:
rich:$1$.FfcK0ns$f1UgiyHQ25wrB/hykCn020:11627:0:99999:7:::
Thereareninefieldsineach/etc/shadowfilerecord:
Theloginnamecorrespondingtotheloginnameinthe/etc/passwdfile
Theencryptedpassword
ThenumberofdayssinceJanuary1,1970,thatthepasswordwaslastchanged
Theminimumnumberofdaysbeforethepasswordcanbechanged
Thenumberofdaysbeforethepasswordmustbechanged
Thenumberofdaysbeforepasswordexpirationthattheuseriswarnedtochangethe
password
Thenumberofdaysafterapasswordexpiresbeforetheaccountwillbedisabled
Thedate(storedasthenumberofdayssinceJanuary1,1970)sincetheuseraccount
wasdisabled
Afieldreservedforfutureuse
Using the shadow password system, the Linux system has much finer control over user
passwords.Itcancontrolhowoftenausermustchangehisorherpasswordandwhento
disabletheaccountifthepasswordhasn’tbeenchanged.
Addinganewuser
TheprimarytoolusedtoaddnewuserstoyourLinuxsystemis useradd.Thiscommand
provides an easy way to create a new user account and set up the user’s HOME directory
structureallatonce.The useraddcommandusesacombinationofsystemdefaultvalues
andcommandlineparameterstodefineauseraccount.Thesystemdefaultsaresetinthe
/etc/default/useradd file. To see the system default values used on your Linux
distribution,entertheuseraddcommandwiththe-Dparameter:
#/usr/sbin/useradd-D
GROUP=100
HOME=/home
INACTIVE=-1
EXPIRE=
SHELL=/bin/bash
SKEL=/etc/skel
CREATE_MAIL_SPOOL=yes
#
Note
SomeLinuxdistributionsplacetheLinuxuserandgrouputilitiesinthe/usr/sbin
directory,whichmaynotbeinyourPATHenvironmentvariable.Ifthat’sthecasein
yourLinuxdistribution,eitheraddthedirectorytoyourPATHorusetheabsolutefile
pathtorunit.
The -D parameter shows what defaults the useradd command uses if you don’t specify
them in the command line when creating a new user account. This example shows the
followingdefaultvalues:
ThenewuserisaddedtoacommongroupwithgroupID100.
ThenewuserhasaHOMEaccountcreatedinthedirectory/home/loginname.
Theaccountcan’tbedisabledwhenthepasswordexpires.
Thenewaccountcan’tbesettoexpireatasetdate.
Thenewaccountusesthebashshellasthedefaultshell.
Thesystemcopiesthecontentsofthe/etc/skeldirectorytotheuser’sHOME
directory.
Thesystemcreatesafileinthemaildirectoryfortheuseraccounttoreceivemail.
The penultimate value is interesting. The useradd command allows an administrator to
createadefault HOMEdirectoryconfigurationandthenusesthatasatemplatetocreatethe
newuser’s HOMEdirectory.Thisallowsyoutoplacedefaultfilesforthesysteminevery
new user’s HOME directory automatically. In the Ubuntu Linux system, the /etc/skel
directoryhasthefollowingfiles:
$ls-al/etc/skel
total32
drwxr-xr-x2rootroot40962010-04-2908:26.
drwxr-xr-x135rootroot122882010-09-2318:49..
-rw-r—r—1rootroot2202010-04-1821:51.bash_logout
-rw-r—r—1rootroot31032010-04-1821:51.bashrc
-rw-r—r—1rootroot1792010-03-2608:31examples.desktop
-rw-r—r—1rootroot6752010-04-1821:51.profile
$
YoushouldrecognizethesefilesfromChapter6.Thesearethestandardstartupfilesfor
thebashshellenvironment.Thesystemautomaticallycopiesthesedefaultfilesintoevery
user’sHOMEdirectoryyoucreate.
Youcantestthisbycreatinganewuseraccountusingthedefaultsystemparametersand
thenlookingattheHOMEdirectoryforthenewuser:
#useradd-mtest
#ls-al/home/test
total24
drwxr-xr-x2testtest40962010-09-2319:01.
drwxr-xr-x4rootroot40962010-09-2319:01..
-rw-r—r—1testtest2202010-04-1821:51.bash_logout
-rw-r—r—1testtest31032010-04-1821:51.bashrc
-rw-r—r—1testtest1792010-03-2608:31examples.desktop
-rw-r—r—1testtest6752010-04-1821:51.profile
#
By default, the useradd command doesn’t create a HOME directory, but the –m command
lineoptiontellsittocreatetheHOMEdirectory.Asyoucanseeintheexample,theuseradd
command created the new HOME directory, using the files contained in the /etc/skel
directory.
Note
Toruntheuseraccountadministrationcommandsinthischapter,youeitherneedto
beloggedinasthespecialrootuseraccountorusethesudocommandtorunthe
commandsastherootuseraccount.
Ifyouwanttooverrideadefaultvalueorbehaviorwhencreatinganewuser,youcando
thatwithcommandlineparameters.TheseareshowninTable7.1.
Table7.1TheuseraddCommandLineParameters
Parameter
-ccomment
Description
Addstexttothenewuser’scommentfield
SpecifiesadifferentnamefortheHOMEdirectoryotherthanthelogin
-dhome_dir
name
-e
Specifiesadate,inYYYY-MM-DDformat,whentheaccountwill
expire_date
expire
Specifiesthenumberofdaysafterapasswordexpireswhentheaccount
-f
willbedisabled.Avalueof0disablestheaccountassoonasthe
inactive_days
passwordexpires;avalueof-1disablesthisfeature.
-g
initial_group
SpecifiesthegroupnameorGIDoftheuser’slogingroup
-Ggroup…
Specifiesoneormoresupplementarygroupstheuserbelongsto
Copiesthe/etc/skeldirectorycontentsintotheuser’sHOMEdirectory
(mustuse-maswell)
Createstheuser’sHOMEdirectory
Doesn’tcreateauser’sHOMEdirectory(usedifthedefaultsettingisto
createone)
Createsanewgroupusingthesamenameastheuser’sloginname
Createsasystemaccount
Specifiesadefaultpasswordfortheuseraccount
Specifiesthedefaultloginshell
SpecifiesauniqueUIDfortheaccount
-k
-m
-M
-n
-r
-ppasswd
-sshell
-uuid
Asyoucansee,youcanoverrideallthesystemdefaultvalueswhencreatinganewuser
accountjustbyusingcommandlineparameters.However,ifyoufindyourselfhavingto
overrideavalueallthetime,it’seasiertojustchangethesystemdefaultvalue.
Youcanchangethesystemdefaultnewuservaluesbyusingthe -Dparameter,alongwith
a parameter representing the value you need to change. These parameters are shown in
Table7.2.
Table7.2TheuseraddChangeDefaultValuesParameters
Parameter
-bdefault_home
Description
Changesthelocationwhereusers’HOMEdirectoriesarecreated
-e
expiration_date
Changestheexpirationdateonnewaccounts
-finactive
-ggroup
-sshell
Changesthenumberofdaysafterapasswordhasexpiredbeforethe
accountisdisabled
ChangesthedefaultgroupnameorGIDused
Changesthedefaultloginshell
Changingthedefaultvaluesisasnap:
#useradd-D-s/bin/tsch
#useradd-D
GROUP=100
HOME=/home
INACTIVE=-1
EXPIRE=
SHELL=/bin/tsch
SKEL=/etc/skel
CREATE_MAIL_SPOOL=yes
#
Now,theuseraddcommandusesthe tschshellasthedefaultloginshellforallnewuser
accountsyoucreate.
Removingauser
Ifyouwanttoremoveauserfromthesystem,the userdelcommandiswhatyouneed.
By default, the userdel command removes only the user information from the
/etc/passwdfile.Itdoesn’tremoveanyfilestheaccountownsonthesystem.
If you use the -r parameter, userdel removes the user’s HOME directory, along with the
user’smaildirectory.However,otherfilesownedbythedeleteduseraccountmaystillbe
onthesystem.Thiscanbeaprobleminsomeenvironments.
Here’sanexampleofusingtheuserdelcommandtoremoveanexistinguseraccount:
#/usr/sbin/userdel-rtest
#ls-al/home/test
ls:cannotaccess/home/test:Nosuchfileordirectory
#
Afterusingthe-rparameter,theuser’sold/home/testdirectorynolongerexists.
Caution
Becarefulwhenusingthe-rparameterinanenvironmentwithlotsofusers.You
neverknowifauserhadimportantfilesstoredinhisorherHOMEdirectorythatare
usedbysomeoneelseoranotherprogram.Alwayscheckbeforeremovingauser’s
HOMEdirectory!
Modifyingauser
Linux provides a few different utilities for modifying the information for existing user
accounts.Table7.3showstheseutilities.
Table7.3UserAccountModificationUtilities
Command
Description
Editsuseraccountfields,aswellasspecifyingprimaryandsecondarygroup
usermod
membership
passwd
Changesthepasswordforanexistinguser
chpasswd
Readsafileofloginnameandpasswordpairs,andupdatesthepasswords
chage
Changesthepassword’sexpirationdate
chfn
Changestheuseraccount’scommentinformation
chsh
Changestheuseraccount’sdefaultshell
Each of these utilities provides a specific function for changing information about user
accounts.Thefollowingsectionsdescribeeachoftheseutilities.
usermod
The usermod command is the most robust of the user account modification utilities. It
providesoptionsforchangingmostofthefieldsinthe /etc/passwdfile.Todothat,you
just need to use the command line parameter that corresponds to the value you want to
change. The parameters are mostly the same as the useradd parameters (such as -c to
changethecommentfield, -etochangetheexpirationdate,and -gtochangethedefault
logingroup).However,acoupleofadditionalparametersmightcomeinhandy:
-lchangestheloginnameoftheuseraccount.
-Llockstheaccountsotheusercan’tlogin.
-pchangesthepasswordfortheaccount.
-Uunlockstheaccountsotheusercanlogin.
The -Lparameterisespeciallyhandy.Usethistolockanaccountsoausercan’tlogin
withouthavingtoremovetheaccountandtheuser’sdata.Toreturntheaccounttonormal,
justusethe-Uparameter.
passwdandchpasswd
Aquickwaytochangejustthepasswordforauseristhepasswdcommand:
#passwdtest
Changingpasswordforusertest.
NewUNIXpassword:
RetypenewUNIXpassword:
passwd:allauthenticationtokensupdatedsuccessfully.
#
Ifyoujustusethe passwdcommandbyitself,itchangesyourownpassword.Anyuserin
the system can change his or her own password, but only the root user can change
someoneelse’spassword.
The -eoptionisahandywaytoforceausertochangethepasswordonthenextlogin.
Thisallowsyoutosettheuser’spasswordtoasimplevalueandforcesthemtochangeit
tosomethingharderthattheycanremember.
If you ever need to do a mass password change for lots of users on the system, the
chpasswdcommandcanbealifesaver.The chpasswdcommandreadsalistofloginname
andpasswordpairs(separatedbyacolon)fromthestandardinput,automaticallyencrypts
thepassword,andsetsitfortheuseraccount.Youcanalsousetheredirectioncommand
toredirectafileofuserid:passwordpairsintothecommand:
#chpasswd<users.txt
#
chsh,chfn,andchage
The chsh, chfn, and chage utilities are specialized for specific account modification
functions.The chshcommandallowsyoutoquicklychangethedefaultloginshellfora
user.Youmustusethefullpathnamefortheshell,andnotjusttheshellname:
#chsh-s/bin/cshtest
Changingshellfortest.
Shellchanged.
#
The chfncommandprovidesastandardmethodforstoringinformationinthecomments
field in the /etc/passwd file. Instead of just inserting random text, such as names or
nicknames,orevenjustleavingthecommentfieldblank,thechfncommandusesspecific
informationusedintheUnix fingercommandtostoreinformationinthecommentfield.
The fingercommandallowsyoutoeasilyfindinformationaboutpeopleonyourLinux
system:
#fingerrich
Login:richName:RichBlum
Directory:/home/richShell:/bin/bash
OnsinceThuSep2018:03(EDT)onpts/0from192.168.1.2
Nomail.
NoPlan.
#
Note
Becauseofsecurityconcerns,manyLinuxsystemadministratorsdisablethefinger
commandontheirsystems,andmanyLinuxdistributionsdon’teveninstallitby
default.
Ifyouusethechfncommandwithnoparameters,itqueriesyoufortheappropriatevalues
toenterintothecommentfield:
#chfntest
Changingfingerinformationfortest.
Name[]:ImaTest
Office[]:DirectorofTechnology
OfficePhone[]:(123)555-1234
HomePhone[]:(123)555-9876
Fingerinformationchanged.
#fingertest
Login:testName:ImaTest
Directory:/home/testShell:/bin/csh
Office:DirectorofTechnologyOfficePhone:(123)555-1234
HomePhone:(123)555-9876
Neverloggedin.
Nomail.
NoPlan.
#
Ifyounowlookattheentryinthe/etc/passwdfile,itlookslikethis:
#greptest/etc/passwd
test:x:504:504:ImaTest,DirectorofTechnology,(123)5551234,(123)555-9876:/home/test:/bin/csh
#
Allthefingerinformationisneatlystoredawayinthe/etc/passwdfileentry.
Finally, the chage command helps you manage the password aging process for user
accounts.Youneedtosetseveralparameterstoindividualvalues,showninTable7.4.
Table7.4ThechageCommandParameters
Parameter
-d
-E
-I
-m
-W
Description
Setsthenumberofdayssincethepasswordwaslastchanged
Setsthedatethepasswordexpires
Setsthenumberofdaysofinactivityafterthepasswordexpirestolockthe
account
Setstheminimumnumberofdaysbetweenpasswordchanges
Setsthenumberofdaysbeforethepasswordexpiresthatawarningmessage
appears
Thechagedatevaluescanbeexpressedusingoneoftwomethods:
AdateinYYYY-MM-DDformat
AnumericalvaluerepresentingthenumberofdayssinceJanuary1,1970
Oneneatfeatureofthe chagecommandisthatitallowsyoutosetanexpirationdatefor
anaccount.Usingthisfeature,youcancreatetemporaryuseraccountsthatautomatically
expireonasetdate,withoutyourhavingtoremembertodeletethem!Expiredaccounts
aresimilartolockedaccounts.Theaccountstillexists,buttheusercan’tloginwithit.
UsingLinuxGroups
User accounts are great for controlling security for individual users, but they aren’t so
goodatallowinggroupsofuserstoshareresources.Toaccomplishthis,theLinuxsystem
usesanothersecurityconcept,calledgroups.
Group permissions allow multiple users to share a common set of permissions for an
object on the system, such as a file, directory, or device (more on that later in the
“DecodingFilePermissions”section).
Linux distributions differ somewhat on how they handle default group memberships.
Some Linux distributions create just one group that contains all the user accounts as
members.YouneedtobecarefulifyourLinuxdistributiondoesthis,becauseyourfiles
may be readable by all other users on the system. Other distributions create a separate
groupaccountforeachusertoprovidealittlemoresecurity.
Each group has a unique GID, which, like UIDs, is a unique numerical value on the
system. Along with the GID, each group has a unique group name. You can use some
grouputilitiestocreateandmanageyourowngroupsontheLinuxsystem.Thissection
discusseshowgroupinformationisstoredandhowtousethegrouputilitiestocreatenew
groupsandmodifyexistinggroups.
The/etc/groupfile
Just like user accounts, group information is stored in a file on the system. The
/etc/group file contains information about each group used on the system. These are
examplesfromatypical/etc/groupfileonaLinuxsystem:
root:x:0:root
bin:x:1:root,bin,daemon
daemon:x:2:root,bin,daemon
sys:x:3:root,bin,adm
adm:x:4:root,adm,daemon
rich:x:500:
mama:x:501:
katie:x:502:
jessica:x:503:
mysql:x:27:
test:x:504:
LikeUIDs,GIDsareassignedusingaspecialformat.Groupsusedforsystemaccountsare
assigned GIDs below 500, and user groups are assigned GIDs starting at 500. The
/etc/groupfileusesfourfields:
Thegroupname
Thegrouppassword
TheGID
Thelistofuseraccountsthatbelongtothegroup
Thegrouppasswordallowsanon-groupmembertotemporarilybecomeamemberofthe
groupbyusingthepassword.Thisfeatureisnotusedallthatcommonly,butitdoesexist.
You should never add users to groups by editing the /etc/group file. Instead, use the
usermod command (discussed earlier in the “Linux Security” section) to add a user
account to a group. Before you can add users to different groups, you must create the
groups.
Note
Thelistofuseraccountsissomewhatmisleading.You’llnoticethatthereareseveral
groupsinthelistthatdon’thaveanyuserslisted.Thisisn’tbecausetheydon’thave
anymembers.Whenauseraccountusesagroupasthedefaultgroupinthe
/etc/passwdfile,theuseraccountdoesn’tappearinthe/etc/groupfileasa
member.Thishascausedconfusionformorethanonesystemadministratoroverthe
years!
Creatingnewgroups
Thegroupaddcommandallowsyoutocreatenewgroupsonyoursystem:
#/usr/sbin/groupaddshared
#tail/etc/group
haldaemon:x:68:
xfs:x:43:
gdm:x:42:
rich:x:500:
mama:x:501:
katie:x:502:
jessica:x:503:
mysql:x:27:
test:x:504:
shared:x:505:
#
When you create a new group, no users are assigned to it by default. The groupadd
commanddoesn’tprovideanoptionforaddinguseraccountstothegroup.Instead,toadd
newusers,usetheusermodcommand:
#/usr/sbin/usermod-Gsharedrich
#/usr/sbin/usermod-Gsharedtest
#tail/etc/group
haldaemon:x:68:
xfs:x:43:
gdm:x:42:
rich:x:500:
mama:x:501:
katie:x:502:
jessica:x:503:
mysql:x:27:
test:x:504:
shared:x:505:rich,test
#
The shared group now has two members, test and rich. The -G parameter in usermod
appendsthenewgrouptothelistofgroupsfortheuseraccount.
Note
Ifyouchangetheusergroupsforanaccountthatiscurrentlyloggedintothesystem,
theusermustlogoutandthenlogbackinforthegroupchangestotakeeffect.
Caution
Becarefulwhenassigninggroupsforuseraccounts.Ifyouusethe-gparameter,the
groupnameyouspecifyreplacesthedefaultgroupfortheuseraccount.The-G
parameteraddsthegrouptothelistofgroupstheuserbelongsto,keepingthedefault
groupintact.
Modifyinggroups
As you can see from the /etc/group file, you don’t need to modify much information
about a group. The groupmod command allows you to change the GID (using the -g
parameter)orthegroupname(usingthe-nparameter)ofanexistinggroup:
#/usr/sbin/groupmod-nsharingshared
#tail/etc/group
haldaemon:x:68:
xfs:x:43:
gdm:x:42:
rich:x:500:
mama:x:501:
katie:x:502:
jessica:x:503:
mysql:x:27:
test:x:504:
sharing:x:505:test,rich
#
Whenchangingthenameofagroup,theGIDandgroupmembersremainthesame,only
thegroupnamechanges.BecauseallsecuritypermissionsarebasedontheGID,youcan
changethenameofagroupasoftenasyouwishwithoutadverselyaffectingfilesecurity.
DecodingFilePermissions
Nowthatyouknowaboutusersandgroups,it’stimetodecodethecrypticfilepermissions
you’ve seen when using the ls command. This section describes how to decipher the
permissionsandwheretheycomefrom.
Usingfilepermissionsymbols
IfyourememberfromChapter3,the lscommandallowsyoutoseethefilepermissions
forfiles,directories,anddevicesontheLinuxsystem:
$ls-l
total68
-rw-rw-r—1richrich502010-09-1307:49file1.gz
-rw-rw-r—1richrich232010-09-1307:50file2
-rw-rw-r—1richrich482010-09-1307:56file3
-rw-rw-r—1richrich342010-09-1308:59file4
-rwxrwxr-x1richrich48822010-09-1813:58myprog
-rw-rw-r—1richrich2372010-09-1813:58myprog.c
drwxrwxr-x2richrich40962010-09-0315:12test1
drwxrwxr-x2richrich40962010-09-0315:12test2
$
Thefirstfieldintheoutputlistingisacodethatdescribesthepermissionsforthefilesand
directories.Thefirstcharacterinthefielddefinesthetypeoftheobject:
-forfiles
dfordirectories
lforlinks
cforcharacterdevices
bforblockdevices
nfornetworkdevices
Afterthat,youseethreesetsofthreecharacters.Eachsetofthreecharactersdefinesan
accesspermissiontriplet:
rforreadpermissionfortheobject
wforwritepermissionfortheobject
xforexecutepermissionfortheobject
If a permission is denied, a dash appears in the location. The three sets relate the three
levelsofsecurityfortheobject:
Theowneroftheobject
Thegroupthatownstheobject
Everyoneelseonthesystem
ThisisbrokendowninFigure7.1.
Figure7.1TheLinuxfilepermissions
Theeasiestwaytodiscussthisistotakeanexampleanddecodethefilepermissionsone
byone:
-rwxrwxr-x1richrich48822010-09-1813:58myprog
Thefilemyproghasthefollowingsetsofpermissions:
rwxforthefileowner(settotheloginnamerich)
rwxforthefilegroupowner(settothegroupnamerich)
r-xforeveryoneelseonthesystem
Thesepermissionsindicatethattheuserloginnamerichcanread,write,andexecutethe
file (considered full permissions). Likewise, members in the group rich can also read,
write,andexecutethefile.However,anyoneelsenotintherichgroupcanonlyreadand
executethefile;the w is replaced with a dash, indicating that write permissions are not
assignedtothissecuritylevel.
Defaultfilepermissions
You may be wondering about where these file permissions come from. The answer is
umask. The umask command sets the default permissions for any file or directory you
create:
$touchnewfile
$ls-alnewfile
-rw-r—r—1richrich0Sep2019:16newfile
$
The touch command created the file using the default permissions assigned to my user
account.Theumaskcommandshowsandsetsthedefaultpermissions:
$umask
0022
$
Unfortunately, the umask command setting isn’t overtly clear, and trying to understand
exactly how it works makes things even muddier. The first digit represents a special
securityfeaturecalledthe stickybit.We’lltalkmoreaboutthatlateroninthischapter
inthe“SharingFiles”section.
The next three digits represent the octal values of the umask for a file or directory. To
understandhowumaskworks,youfirstneedtounderstandoctalmodesecuritysettings.
Octalmodesecuritysettingstakethethree rwxpermissionvaluesandconvertthemintoa
3-bitbinaryvalue,representedbyasingleoctalvalue.Inthebinaryrepresentation,each
positionisabinarybit.Thus,ifthereadpermissionistheonlypermissionset,thevalue
becomes r—, relating to a binary value of 100, indicating the octal value of 4. Table 7.5
showsthepossiblecombinationsyou’llruninto.
Table7.5LinuxFilePermissionCodes
Permissions Binary Octal
Description
–
000
0
Nopermissions
—x
001
1
Execute-onlypermission
-w010
2
Write-onlypermission
-wx
011
3
Writeandexecutepermissions
r—
100
4
Read-onlypermission
r-x
101
5
Readandexecutepermissions
rw110
6
Readandwritepermissions
rwx
111
7 Read,write,andexecutepermissions
Octal mode takes the octal permissions and lists three of them in order for the three
securitylevels(user,group,andeveryone).Thus,theoctalmodevalue664representsread
andwritepermissionsfortheuserandgroup,butread-onlypermissionforeveryoneelse.
Nowthatyouknowaboutoctalmodepermissions,the umaskvaluebecomesevenmore
confusing.TheoctalmodeshownforthedefaultumaskonmyLinuxsystemis0022,but
thefileIcreatedhadanoctalmodepermissionof644.Howdidthathappen?
Theumaskvalueisjustthat,amask.Itmasksoutthepermissionsyoudon’twanttogive
tothesecuritylevel.Nowwehavetodiveintosomeoctalarithmetictofigureouttherest
ofthestory.
The umask value is subtracted from the full permission set for an object. The full
permission for a file is mode 666 (read/write permission for all), but for a directory it’s
777(read/write/executepermissionforall).
Thus, in the example, the file starts out with permissions 666, and the umask of 022 is
applied,leavingafilepermissionof644.
The umask value is normally set in the /etc/profile startup file in most Linux
distributions(seeChapter6),butsomeprefertosetitinthe/etc/login.defsfile(suchas
inUbuntu).Youcanspecifyadifferentdefaultumasksettingusingtheumaskcommand:
$umask026
$touchnewfile2
$ls-lnewfile2
-rw-r–—1richrich0Sep2019:46newfile2
$
Bysettingtheumaskvalueto026,thedefaultfilepermissionsbecome640,sothenew
filenowisrestrictedtoread-onlyforthegroupmembers,andeveryoneelseonthesystem
hasnopermissionstothefile.
Theumaskvaluealsoappliestomakingnewdirectories:
$mkdirnewdir
$ls-l
drwxr-x—x2richrich4096Sep2020:11newdir/
$
Becausethedefaultpermissionsforadirectoryare777,theresultingpermissionsfromthe
umaskaredifferentfromthoseofanewfile.The026umaskvalueissubtractedfrom777,
leavingthe751directorypermissionsetting.
ChangingSecuritySettings
Ifyou’vealreadycreatedafileordirectoryandneedtochangethesecuritysettingsonit,
Linuxhasafewdifferentutilitiesavailableforthis.Thissectionshowsyouhowtochange
the existing permissions, the default owner, and the default group settings for a file or
directory.
Changingpermissions
The chmodcommandallowsyoutochangethesecuritysettingsforfilesanddirectories.
Theformatofthechmodcommandis:
chmodoptionsmodefile
The modeparameterallowsyoutosetthesecuritysettingsusingeitheroctalorsymbolic
mode.Theoctalmodesettingsareprettystraightforward;justusethestandardthree-digit
octalcodeyouwantthefiletohave:
$chmod760newfile
$ls-lnewfile
-rwxrw–-1richrich0Sep2019:16newfile
$
The octal file permissions are automatically applied to the file indicated. The symbolic
modepermissionsarenotsoeasytoimplement.
Instead of using the normal string of three sets of three characters, the chmod command
takes a different approach. The following is the format for specifying a permission in
symbolicmode:
[ugoa…][[+-=][rwxXstugo…]
Makesperfectlygoodsense,doesn’tit?Thefirstgroupofcharactersdefinestowhomthe
newpermissionsapply:
ufortheuser
gforthegroup
oforothers(everyoneelse)
aforalloftheabove
Next,asymbolisusedtoindicatewhetheryouwanttoaddthepermissiontotheexisting
permissions (+), subtract the permission from the existing permission (−), or set the
permissionstothevalue(=).
Finally,thethirdsymbolisthepermissionusedforthesetting.Youmaynoticethatthere
aremorethanthenormalrwxvalueshere.Thesearetheadditionalsettings:
Xassignsexecutepermissionsonlyiftheobjectisadirectoryorifitalreadyhad
executepermissions.
ssetstheUIDorGIDonexecution.
tsavesprogramtext.
usetsthepermissionstotheowner’spermissions.
gsetsthepermissionstothegroup’spermissions.
osetsthepermissionstotheother’spermissions.
Usingthesepermissionslookslikethis:
$chmodo+rnewfile
$ls-lFnewfile
-rwxrw-r—1richrich0Sep2019:16newfile*
$
The o+r entry adds the read permission to whatever permissions the everyone security
levelalreadyhad.
$chmodu-xnewfile
$ls-lFnewfile
-rw-rw-r—1richrich0Sep2019:16newfile
$
The u-xentryremovestheexecutepermissionthattheuseralreadyhad.Notethatthe –F
optionforthe lscommandindicateswhetherafilehasexecutionpermissionsbyadding
anasterisktothefilename.
Theoptionsparametersprovideafewadditionalfeaturestoaugmentthebehaviorofthe
chmod command. The -R parameter performs the file and directory changes recursively.
Youcanusewildcardcharactersforthefilenamespecified,changingthepermissionson
multiplefileswithjustonecommand.
Changingownership
Sometimes, you need to change the owner of a file, such as when someone leaves an
organization or a developer creates an application that needs to be owned by a system
accountwhenit’sinproduction.Linuxprovidestwocommandsfordoingthat.The chown
commandmakesiteasytochangetheownerofafile,andthechgrpcommandallowsyou
tochangethedefaultgroupofafile.
Theformatofthechowncommandis:
chownoptionsowner[.group]file
YoucanspecifyeithertheloginnameorthenumericUIDforthenewownerofthefile:
#chowndannewfile
#ls-lnewfile
-rw-rw-r—1danrich0Sep2019:16newfile
#
Simple.Thechowncommandalsoallowsyoutochangeboththeuserandgroupofafile:
#chowndan.sharednewfile
#ls-lnewfile
-rw-rw-r—1danshared0Sep2019:16newfile
#
Ifyoureallywanttogettricky,youcanjustchangethedefaultgroupforafile:
#chown.richnewfile
#ls-lnewfile
-rw-rw-r—1danrich0Sep2019:16newfile
#
Finally, if your Linux system uses individual group names that match user login names,
youcanchangebothwithjustoneentry:
#chowntest.newfile
#ls-lnewfile
-rw-rw-r—1testtest0Sep2019:16newfile
#
Thechowncommandusesafewdifferentoptionparameters.The-Rparameterallowsyou
tomakechangesrecursivelythroughsubdirectoriesandfiles,usingawildcardcharacter.
The -hparameteralsochangestheownershipofanyfilesthataresymbolicallylinkedto
thefile.
Note
Onlytherootusercanchangetheownerofafile.Anyusercanchangethedefault
groupofafile,buttheusermustbeamemberofthegroupsthefileischangedfrom
andto.
The chgrpcommandprovidesaneasywaytochangejustthedefaultgroupforafileor
directory:
$chgrpsharednewfile
$ls-lnewfile
-rw-rw-r—1richshared0Sep2019:16newfile
$
Theuseraccountmustownthefile,andbeamemberofthenewgroupaswelltobeable
tochangethegroup.Nowanymemberinthesharedgroupcanwritetothefile.Thisis
onewaytosharefilesonaLinuxsystem.However,sharingfilesamongagroupofpeople
onthesystemcangettricky.Thenextsectiondiscusseshowtodothis.
SharingFiles
Asyou’veprobablyalreadyfiguredout,creatinggroupsisthewaytoshareaccesstofiles
ontheLinuxsystem.However,foracompletefile-sharingenvironment,thingsaremore
complicated.
As you’ve already seen in the “Decoding File Permissions” section, when you create a
new file, Linux assigns the file permissions of the new file using your default UID and
GID.Toallowothersaccesstothefile,youneedtoeitherchangethesecuritypermissions
for the everyone security group or assign the file a different default group that contains
otherusers.
This can be a pain in a large environment if you want to create and share documents
amongseveralpeople.Fortunately,there’sasimplesolutionforhowtosolvethisproblem.
TherearethreeadditionalbitsofinformationthatLinuxstoresforeachfileanddirectory:
Thesetuserid(SUID):Whenafileisexecutedbyauser,theprogramrunsunder
thepermissionsofthefileowner.
Thesetgroupid(SGID):Forafile,theprogramrunsunderthepermissionsofthe
filegroup.Foradirectory,newfilescreatedinthedirectoryusethedirectorygroup
asthedefaultgroup.
Thestickybit:Thefileremains(sticks)inmemoryaftertheprocessends.
TheSGIDbitisimportantforsharingfiles.ByenablingtheSGIDbit,youcanforceall
newfilescreatedinashareddirectorytobeownedbythedirectory’sgroupandnowthe
individualuser’sgroup.
The SGID is set using the chmod command. It’s added to the beginning of the standard
three-digit octal value (making a four-digit octal value), or you can use the symbol sin
symbolicmode.
If you’re using octal mode, you’ll need to know the arrangement of the bits, shown in
Table7.6.
Table7.6ThechmodSUID,SGID,andStickyBitOctalValues
Binary Octal
Description
000
0
Allbitsarecleared.
001
1
Thestickybitisset.
010
2
TheSGIDbitisset.
011
3 TheSGIDandstickybitsareset.
100
4
TheSUIDbitisset.
101
5 TheSUIDandstickybitsareset.
110
6 TheSUIDandSGIDbitsareset.
111
7
Allbitsareset.
So,tocreateashareddirectorythatalwayssetsthedirectorygroupforallnewfiles,all
youneedtodoissettheSGIDbitforthedirectory:
$mkdirtestdir
$ls-l
drwxrwxr-x2richrich4096Sep2023:12testdir/
$chgrpsharedtestdir
$chmodg+stestdir
$ls-l
drwxrwsr-x2richshared4096Sep2023:12testdir/
$umask002
$cdtestdir
$touchtestfile
$ls-l
total0
-rw-rw-r—1richshared0Sep2023:13testfile
$
The first step is to create a directory that you want to share using the mkdircommand.
Next,usethechgrpcommandtochangethedefaultgroupforthedirectorytoagroupthat
containsthememberswhoneedtosharefiles(youmustbeamemberofthatgroupfor
thistowork).Finally,settheSGIDbitforthedirectorytoensurethatanyfilescreatedin
thedirectoryusethesharedgroupnameasthedefaultgroup.
For this environment to work properly, all the group members must have their umask
valuessettomakefileswritablebygroupmembers.Intheprecedingexample,the umask
ischangedto002sothefilesarewritablebythegroup.
Afterallthat’sdone,anymemberofthegroupcangototheshareddirectoryandcreatea
new file. As expected, the new file uses the default group of the directory, not the user
account’sdefaultgroup.Nowanyuserinthesharedgroupcanaccessthisfile.
Summary
This chapter discussed the command line commands you need to know to manage the
Linuxsecurityonyoursystem.LinuxusesasystemofuserIDsandgroupIDstoprotect
access to files, directories, and devices. Linux stores information about user accounts in
the /etc/passwd file and information about groups in the /etc/group file. Each user is
assignedauniquenumericuserID,alongwithatextloginnametoidentifytheuserinthe
system. Groups are also assigned unique numerical group IDs and text group names. A
groupcancontainoneormoreuserstoallowedsharedaccesstosystemresources.
Several commands are available for managing user accounts and groups. The useradd
commandallowsyoutocreatenewuseraccounts,andthegroupaddcommandallowsyou
to create new group accounts. To modify an existing user account, use the usermod
command.Similarly,usethegroupmodcommandtomodifygroupaccountinformation.
Linux uses a complicated system of bits to determine access permissions for files and
directories.Eachfilecontainsthreesecuritylevelsofprotection:thefile’sowner,adefault
groupthathasaccesstothefile,andalevelforeveryoneelseonthesystem.Eachsecurity
level is defined by three access bits: read, write, and execute. The combination of three
bitsisoftenreferredtobythesymbolsrwx,forread,write,andexecute.Ifapermissionis
denied,itssymbolisreplacedwithadash(suchasr—forread-onlypermission).
The symbolic permissions are often referred to as octal values, with the three bits
combinedintooneoctalvalueandthreeoctalvaluesrepresentingthethreesecuritylevels.
Usetheumaskcommandtosetthedefaultsecuritysettingsforfilesanddirectoriescreated
on the system. The system administrator normally sets a default umask value in the
/etc/profile file, but you can use the umask command to change your umask value at
anytime.
Use the chmod command to change security settings for files and directories. Only the
file’s owner can change permissions for a file or directory. However, the root user can
changethesecuritysettingsforanyfileordirectoryonthesystem.Youcanusethechown
andchgrpcommandstochangethedefaultownerandgroupofthefile.
The chapter closed with a discussion on how to use the set GID bit to create a shared
directory.TheSGIDbitforcesanynewfilesordirectoriescreatedinadirectorytousethe
default group name of the parent directory, not that of the user who created them. This
providesaneasywaytosharefilesbetweenusersonthesystem.
Nowthatyou’reuptospeedwithfilepermissions,it’stimetotakeacloserlookathowto
workwiththeactualfilesysteminLinux.Thenextchaptershowsyouhowtocreatenew
partitionsinLinuxfromthecommandlineandthenhowtoformatthenewpartitionsso
thattheycanbeusedintheLinuxvirtualdirectory.
Chapter8
ManagingFilesystems
InThisChapter
1. Understandingfilesystembasics
2. Exploringjournalingandcopy-on-writefilesystems
3. Managingfilesystems
4. Investigatingthelogicalvolumelayout
5. UsingtheLinuxLogicalVolumeManager
Whenyou’reworkingwithyourLinuxsystem,oneofthedecisionsyou’llneedto
makeiswhatfilesystemtouseforthestoragedevices.MostLinuxdistributions
kindlyprovideadefaultfilesystemforyouatinstallationtime,andmostbeginning
Linuxusersjustuseitwithoutgivingthetopicanotherthought.
Althoughusingthedefaultfilesystemchoiceisn’tnecessarilyabadthing,sometimes
ithelpstoknowtheotheroptionsavailabletoyou.Thischapterdiscussesthe
differentfilesystemoptionsyouhaveavailableintheLinuxworldandshowsyou
howtocreateandmanagethemfromtheLinuxcommandline.
ExploringLinuxFilesystems
Chapter 3 discussed how Linux uses a filesystem to store files and folders on a storage
device.ThefilesystemprovidesawayforLinuxtobridgethegapbetweentheonesand
zeroes stored in the hard drive and the files and folders you work with in your
applications.
Linuxsupportsseveraltypesoffilesystemstomanagefilesandfolders.Eachfilesystem
implements the virtual directory structure on storage devices using slightly different
features. This section walks you through the strengths and weaknesses of the more
commonfilesystemsusedintheLinuxenvironment.
UnderstandingthebasicLinuxfilesystems
TheoriginalLinuxsystemusedasimplefilesystemthatmimickedthefunctionalityofthe
Unixfilesystem.Thissectiondiscussestheevolutionofthatfilesystem.
LookingattheextFilesystem
TheoriginalfilesystemintroducedwiththeLinuxoperatingsystemiscalledtheextended
filesystem(orjustextforshort).ItprovidesabasicUnix-likefilesystemforLinux,using
virtual directories to handle physical devices, and storing data in fixed-length blocks on
thephysicaldevices.
Theextfilesystemusesasystemcalledinodestotrackinformationaboutthefilesstored
inthevirtualdirectory.Theinodesystemcreatesaseparatetableoneachphysicaldevice,
calledtheinodetable,tostorefileinformation.Eachstoredfileinthevirtualdirectoryhas
anentryintheinodetable.Theextendedpartofthenamecomesfromtheadditionaldata
thatittracksoneachfile,whichconsistsoftheseitems:
Thefilename
Thefilesize
Theownerofthefile
Thegroupthefilebelongsto
Accesspermissionsforthefile
Pointerstoeachdiskblockthatcontainsdatafromthefile
Linux references each inode in the inode table using a unique number (called the inode
number),assignedbythefilesystemasdatafilesarecreated.Thefilesystemusestheinode
numbertoidentifythefileratherthanhavingtousethefullfilenameandpath.
Lookingattheext2Filesystem
The original ext filesystem had quite a few limitations, such as restraining files to only
2GB in size. Not too long after Linux was first introduced, the ext filesystem was
upgradedtocreatethesecondextendedfilesystem,calledext2.
As you can guess, the ext2 filesystem is an expansion of the basic abilities of the ext
filesystem,butmaintainsthesamestructure.Theext2filesystemexpandstheinodetable
formattotrackadditionalinformationabouteachfileonthesystem.
Theext2inodetableaddsthecreated,modified,andlastaccessedtimevaluesforfilesto
help system administrators track file access on the system. The ext2 filesystem also
increasesthemaximumfilesizeallowedto2TB(theninlaterversionsofext2,thatwas
increasedto32TB)tohelpaccommodatelargefilescommonlyfoundindatabaseservers.
In addition to expanding the inode table, the ext2 filesystem also changed the way in
whichfilesarestoredinthedatablocks.Acommonproblemwiththeextfilesystemwas
thatasafileiswrittentothephysicaldevice,theblocksusedtostorethedatatendtobe
scatteredthroughoutthedevice(calledfragmentation).Fragmentationofdatablockscan
reducethefilesystemperformance,becauseittakeslongertosearchthestoragedeviceto
accessalltheblocksforaspecificfile.
Theext2filesystemhelpsreducefragmentationbyallocatingdiskblocksingroupswhen
you save a file. By grouping the data blocks for a file, the filesystem doesn’t have to
searchalloverthephysicaldeviceforthedatablockstoreadthefile.
Theext2filesystemwasthedefaultfilesystemusedinLinuxdistributionsformanyyears,
but it, too, had its limitations. The inode table, although a nice feature that allows the
filesystemtotrackadditionalinformationaboutfiles,cancauseproblemsthatcanbefatal
tothesystem.Eachtimethefilesystemstoresorupdatesafile,itmustmodifytheinode
tablewiththenewinformation.Theproblemisthatthisisn’talwaysafluidaction.
Ifsomethingshouldhappentothecomputersystembetweenthefilebeingstoredandthe
inode table being updated, the two would become out of sync. The ext2 filesystem is
notoriousforeasilybecomingcorruptedduetosystemcrashesandpoweroutages.Evenif
the file data is stored just fine on the physical device, if the inode table entry isn’t
completed,theext2filesystemdoesn’tevenknowthatthefileexisted!
Itwasn’tlongbeforedeveloperswereexploringadifferentavenueofLinuxfilesystems.
Understandingjournalingfilesystems
Journaling filesystems provide a new level of safety to the Linux system. Instead of
writing data directly to the storage device and then updating the inode table, journaling
filesystemswritefilechangesintoatemporaryfile(calledthejournal)first.Afterdatais
successfullywrittentothestoragedeviceandtheinodetable,thejournalentryisdeleted.
Ifthesystemshouldcrashorsufferapoweroutagebeforethedatacanbewrittentothe
storagedevice,thejournalingfilesystemjustreadsthroughthejournalfileandprocesses
anyuncommitteddataleftover.
Linuxcommonlyusesthreedifferentmethodsofjournaling,eachwithdifferentlevelsof
protection.TheseareshowninTable8.1.
Table8.1JournalingFilesystemMethods
Method
Data
mode
Description
Bothinodeandfiledataarejournaled.Lowriskoflosingdata,butpoor
performance.
Ordered
mode
Onlyinodedataiswrittentothejournal,butnotremoveduntilfiledatais
successfullywritten.Goodcompromisebetweenperformanceandsafety.
Writeback Onlyinodedataiswrittentothejournal,nocontroloverwhenthefiledatais
mode
written.Higherriskoflosingdata,butstillbetterthannotusingjournaling.
Thedatamodejournalingmethodisbyfarthesafestforprotectingdata,butitisalsothe
slowest.Allthedatawrittentoastoragedevicemustbewrittentwice,oncetothejournal
and again to the actual storage device. This can cause poor performance, especially for
systemsthatdolotsofdatawriting.
Over the years, a few different journaling filesystems have appeared in Linux. The
followingsectionsdescribethepopularLinuxjournalingfilesystemsavailable.
Lookingattheext3Filesystem
Theext3filesystemwasaddedtotheLinuxkernelin2001,andupuntilrecentlywasthe
defaultfilesystemusedbyjustaboutallLinuxdistributions.Itusesthesameinodetable
structureastheext2filesystem,butaddsajournalfiletoeachstoragedevicetojournalthe
datawrittentothestoragedevice.
Bydefault,theext3filesystemusestheorderedmodemethodofjournaling,onlywriting
the inode information to the journal file, but not removing it until the data blocks have
been successfully written to the storage device. You can change the journaling method
usedintheext3filesystemtoeitherdataorwritebackmodeswithasimplecommandline
optionwhencreatingthefilesystem.
Althoughtheext3filesystemaddedbasicjournalingtotheLinuxfilesystem,itstilllacked
a few things. For example, the ext3 filesystem doesn’t provide any recovery from
accidentaldeletionoffiles,nobuilt-indatacompressionisavailable(althoughapatchcan
beinstalledseparatelythatprovidesthisfeature),andtheext3filesystemdoesn’tsupport
encryptingfiles.Forthosereasons,developersintheLinuxprojectchosetocontinuework
onimprovingtheext3filesystem.
Lookingattheext4Filesystem
The result of expanding the ext3 filesystem was (as you probably guessed) the ext4
filesystem.Theext4filesystemwasofficiallysupportedintheLinuxkernelin2008andis
nowthedefaultfilesystemusedinpopularLinuxdistributions,suchasUbuntu.
Inadditiontosupportingcompressionandencryption,theext4filesystemalsosupportsa
featurecalledextents.Extentsallocatespaceonastoragedeviceinblocksandonlystore
thestartingblocklocationintheinodetable.Thishelpssavespaceintheinodetableby
nothavingtolistallthedatablocksusedtostoredatafromthefile.
Theext4filesystemalsoincorporatesblockpreallocation.Ifyouwanttoreservespaceon
a storage device for a file that you know will grow in size, with the ext4 filesystem it’s
possibletoallocatealltheexpectedblocksforthefile,notjusttheblocksthatphysically
exist.Theext4filesystemfillsinthereserveddatablockswithzeroesandknowsnotto
allocatethemforanyotherfile.
LookingattheReiserFilesystem
In 2001, Hans Reiser created the first journaling filesystem for Linux, called ReiserFS.
TheReiserFSfilesystemsupportsonlywritebackjournalingmode,writingonlytheinode
tabledatatothejournalfile.Becauseitwritesonlytheinodetabledatatothejournal,the
ReiserFSfilesystemisoneofthefasterLinuxjournalingfilesystems.
TwointerestingfeaturesincorporatedintotheReiserFSfilesystemarethatyoucanresize
anexistingfilesystemwhileit’sstillactiveandthatitusesatechniquecalledtailpacking,
which stuffs data from one file into empty space in a data block from another file. The
active filesystem resizing feature is great if you have to expand an already created
filesystemtoaccommodatemoredata.
TheReiserFSdevelopmentteambeganworkingonanewversioncalledReiser4in2004.
The Reiser4 filesystem has several improvements over ResierFS, including extremely
efficient handling of small files. However, most current mainstream Linux distributions
don’tusetheReiser4filesystem.Yet,youmaystillrunintoaLinuxsystemthatemploys
it.
LookingattheJournaledFilesystem
Possiblyoneoftheoldestjournalingfilesystemsaround,theJournaledFileSystem(JFS)
was developed by IBM in 1990 for its AIX flavor of Unix. However, it wasn’t until its
secondversionthatitwasportedtotheLinuxenvironment.
Note
TheofficialIBMnameofthesecondversionoftheJFSfilesystemisJFS2,butmost
LinuxsystemsrefertoitasjustJFS.
TheJFSfilesystemusestheorderedjournalingmethod,storingonlytheinodetabledata
inthejournal,butnotremovingituntiltheactualfiledataiswrittentothestoragedevice.
This method is a compromise between the speed of the Reiser4 and the integrity of the
datamodejournalingmethod.
TheJFSfilesystemusesextent-basedfileallocation,allocatingagroupofblocksforeach
file written to the storage device. This method provides for less fragmentation on the
storagedevice.
OutsideoftheIBMLinuxofferings,theJFSfilesystemisn’tpopularlyused,butyoumay
runintoitinyourLinuxjourney.
LookingattheXFSFilesystem
The XFS journaling filesystem is yet another filesystem originally created for a
commercial Unix system that made its way into the Linux world. Silicon Graphics
Incorporated(SGI)originallycreatedXFSin1994foritscommercialIRIXUnixsystem.
It was released to the Linux environment for common use in 2002. The XFS filesystem
has recently become more popular and is used as the default filesystem in mainstream
Linuxdistributions,suchasRHEL.
The XFS filesystem uses the writeback mode of journaling, which provides high
performancebutdoesintroduceanamountofriskbecausetheactualdataisn’tstoredin
thejournalfile.TheXFSfilesystemalsoallowsonlineresizingofthefilesystem,similar
totheReiser4filesystem,exceptXFSfilesystemscanonlybeexpandedandnotshrunk.
Understandingthecopy-on-writefilesystems
Withjournaling,youmustchoosebetweensafetyandperformance.Althoughdatamode
journalingprovidesthehighestsafety,performancesuffersbecausebothinodeanddatais
journaled. With writeback mode journaling, performance is acceptable, but safety is
compromised.
For filesystems, an alternative to journaling is a technique called copy-on-write(COW).
COW offers both safety and performance via snapshots. For modifying data, a cloneor
writable-snapshotisused.Insteadofwritingmodifieddataovercurrentdata,themodified
data is put in a new filesystem location. Even when data modification is completed, the
olddataisneveroverwritten.
COWfilesystemsaregaininginpopularity.Twoofthemostpopular,BtrfsandZFS,are
brieflyreviewedinthefollowingsections.
LookingattheZFSFilesystem
The COW filesystem ZFS was developed in 2005 by Sun Microsystems for the
OpenSolaris operating system. It began being ported to Linux in 2008 and was finally
availableforLinuxproductionusein2012.
ZFSisastablefilesystemandcompeteswellagainstResier4,Btrfs,andext4.Itsbiggest
detractoristhatZFSdoesnothaveaGPLlicense.TheOpenZFSprojectwaslaunchedin
2013, which may help to change this situation. However, it’s possible that until a GPL
licenseisobtained,ZFSwillneverbeadefaultLinuxfilesystem.
LookingattheBtrfsFilesystem
The COW newcomer is the Btrfs filesystem, also called the B-tree filesystem. Oracle
started development on Btrfs in 2007. It was based on many of Reiser4’s features, but
offeredimprovementsinreliability.Additionaldeveloperseventuallyjoinedinandhelped
Btrfsquicklyrisetowardthetopofthepopularfilesystemslist.Thispopularityisdueto
stability, ease of use, as well as the ability to dynamically resize a mounted filesystem.
TheopenSUSELinuxdistributionrecentlyestablishedBtrfsasitsdefaultfilesystem.Itis
also offered in other Linux distributions, such as RHEL, although not as the default
filesystem.
WorkingwithFilesystems
Linuxprovidesafewdifferentutilitiesthatmakeiteasiertoworkwithfilesystemsfrom
thecommandline.Youcanaddnewfilesystemsorchangeexistingfilesystemsfromthe
comfort of your own keyboard. This section walks you through the commands for
interactingwithfilesystemsfromacommandlineenvironment.
Creatingpartitions
Tostartout,youneedtocreateapartitiononthestoragedevicetocontainthefilesystem.
The partition can be an entire disk or a subset of a disk that contains a portion of the
virtualdirectory.
Thefdiskutilityisusedtohelpyouorganizepartitionsonanystoragedeviceinstalledon
the system. The fdisk command is an interactive program that allows you to enter
commandstowalkthroughthestepsofpartitioningaharddrive.
Tostartthe fdiskcommand,youneedtospecifythedevicenameofthestoragedevice
you want to partition and you need to have superuser privileges. When you don’t have
superuserprivilegesandattempttouse fdisk,you’llreceivesomesortoferrormessage,
likethisone:
$fdisk/dev/sdb
Unabletoopen/dev/sdb
$
Note
Sometimes,thehardestpartofcreatinganewdiskpartitionistryingtofindthe
physicaldiskonyourLinuxsystem.Linuxusesastandardformatforassigning
devicenamestoharddrives,butyouneedtobefamiliarwiththeformat.Forolder
IDEdrives,Linuxuses/dev/hdx,wherexisaletterbasedontheorderthedriveis
detected(aforthefirstdrive,bforthesecond,andsoon).ForboththenewerSATA
drivesandSCSIdrives,Linuxuses/dev/sdx,wherexisaletterbasedontheorder
thedriveisdetected(again,aforthefirstdrive,bforthesecond,andsoon).It’s
alwaysagoodideatodouble-checktomakesureyouarereferencingthecorrect
drivebeforeformattingthepartition!
If you do have superuser privileges and the correct device name, the fdisk command
allowsyouentranceintotheutilityasdemonstratedhereonaCentOSdistribution:
$sudofdisk/dev/sdb
[sudo]passwordforChristine:
DevicecontainsneitheravalidDOSpartitiontable,
norSun,SGIorOSFdisklabel
BuildinganewDOSdisklabelwithdiskidentifier0xd3f759b5.
Changeswillremaininmemoryonly
untilyoudecidetowritethem.
Afterthat,ofcourse,thepreviouscontentwon’tberecoverable.
Warning:invalidflag0x0000ofpartitiontable4will
becorrectedbyw(rite)
[…]
Command(mforhelp):
Tip
Ifthisisthefirsttimeyou’repartitioningthestoragedevice,fdiskgivesyoua
warningthatapartitiontableisnotonthedevice.
The fdisk interactive command prompt uses single letter commands to instruct fdisk
whattodo.Table8.2showsthecommandsavailableatthefdiskcommandprompt.
Table8.2ThefdiskCommands
Command
a
b
c
d
l
m
n
o
p
q
s
t
u
v
w
x
Description
Togglesaflagindicatingifthepartitionisbootable
EditsthedisklabelusedbyBSDUnixsystems
TogglestheDOScompatibilityflag
Deletesthepartition
Liststheavailablepartitiontypes
Displaysthecommandoptions
Addsanewpartition
CreatesaDOSpartitiontable
Displaysthecurrentpartitiontable
Quitswithoutsavingchanges
CreatesanewdisklabelforSunUnixsystems
ChangesthepartitionsystemID
Changesthestorageunitsused
Verifiesthepartitiontable
Writesthepartitiontabletothedisk
Advancedfunctions
Althoughthislistmaylookintimidating,usuallyyouneedjustafewbasiccommandsin
day-to-daywork.
Forstarters,youcandisplaythedetailsofastoragedeviceusingthepcommand:
Command(mforhelp):p
Disk/dev/sdb:5368MB,5368709120bytes
255heads,63sectors/track,652cylinders
Units=cylindersof16065*512=8225280bytes
Sectorsize(logical/physical):512bytes/512bytes
I/Osize(minimum/optimal):512bytes/512bytes
Diskidentifier:0x11747e88
DeviceBootStartEndBlocksIdSystem
Command(mforhelp):
Theoutputshowsthatthestoragedevicehas5368MBofspaceonit(5GB).Thelisting
under the storage device details shows whether there are any existing partitions on the
device. The listing in this example doesn’t show any partitions, so the device is not
partitionedyet.
Next,you’llwanttocreateanewpartitiononthestoragedevice.Usethe ncommandfor
that:
Command(mforhelp):n
Commandaction
eextended
pprimarypartition(1-4)
p
Partitionnumber(1-4):1
Firstcylinder(1-652,default1):1
Lastcylinder,+cylindersor+size{K,M,G}(1-652,default652):+2G
Command(mforhelp):
Partitions can be created as either a primarypartition or an extendedpartition. Primary
partitions can be formatted with a filesystem directly, whereas extended partitions can
onlycontainotherprimarypartitions.Thereasonforextendedpartitionsisthattherecan
onlybefourpartitionsonasinglestoragedevice.Youcanextendthatbycreatingmultiple
extended partitions and then creating primary partitions inside the extended partitions.
This example creates a primary storage device, assigns it partition number 1, and then
allocates 2GB of the storage device space to it. You can see the results using the p
commandagain:
Command(mforhelp):p
Disk/dev/sdb:5368MB,5368709120bytes
255heads,63sectors/track,652cylinders
Units=cylindersof16065*512=8225280bytes
Sectorsize(logical/physical):512bytes/512bytes
I/Osize(minimum/optimal):512bytes/512bytes
Diskidentifier:0x029aa6af
DeviceBootStartEndBlocksIdSystem
/dev/sdb112622104483+83Linux
Command(mforhelp):
Nowintheoutputthere’sapartitionshownonthestoragedevice(called/dev/sdb1).The
IdentrydefineshowLinuxtreatsthepartition.fdiskallowsyoutocreatelotsofpartition
types. Using the l command lists the different types available. The default is type 83,
which defines a Linux filesystem. If you want to create a partition for a different
filesystem(suchasaWindowsNTFSpartition),justselectadifferentpartitiontype.
Youcanrepeattheprocesstoallocatetheremainingspaceonthestoragedevicetoanother
Linuxpartition.Afteryou’vecreatedthepartitionsyouwant,usethe wcommandtosave
thechangestothestoragedevice:
Command(mforhelp):w
Thepartitiontablehasbeenaltered!
Callingioctl()tore-readpartitiontable.
Syncingdisks.
$
Thestoragedevicepartitioninformationwaswrittentothepartitiontable,andLinuxwas
informedofthenewpartitionviatheioctl()call.Nowthatyouhavesetupapartitionon
thestoragedevice,you’rereadytoformatitwithaLinuxfilesystem.
Tip
Somedistributionsandolderdistributionversionsdonotautomaticallyinformyour
Linuxsystemofanewpartitionafteritismade.Inthiscase,youneedtouseeither
thepartprobeorhdparmcommand(seetheirmanpages),orrebootyoursystemsoit
readstheupdatedpartitiontable
Creatingafilesystem
Beforeyoucanstoredataonthepartition,youmustformatitwithafilesystemsoLinux
canuseit.Eachfilesystemtypeusesitsowncommandlineprogramtoformatpartitions.
Table8.3liststheutilitiesusedforthedifferentfilesystemsdiscussedinthischapter.
Table8.3CommandLineProgramstoCreateFilesystems
Utility
mkefs
mke2fs
mkfs.ext3
mkfs.ext4
mkreiserfs
jfs_mkfs
mkfs.xfs
mkfs.zfs
mkfs.btrfs
Purpose
Createsanextfilesystem
Createsanext2filesystem
Createsanext3filesystem
Createsanext4filesystem
CreatesaReiserFSfilesystem
CreatesaJFSfilesystem
CreatesanXFSfilesystem
CreatesaZFSfilesystem
CreatesaBtrfsfilesystem
Not all filesystem utilities are installed by default. To determine whether you have a
particularfilesystemutility,usethetypecommand:
$typemkfs.ext4
mkfs.ext4is/sbin/mkfs.ext4
$
$typemkfs.btrfs
-bash:type:mkfs.btrfs:notfound
$
TheprecedingexampleonanUbuntusystemshowsthatthemkfs.ext4utilityisavailable.
However,theBtrfsutilityisnot.SeeChapter9onhowtoinstalladditionalsoftwareand
utilitiesonyourLinuxdistribution.
Each filesystem utility command has lots of command line options that allow you to
customizejusthowthefilesystemiscreatedinthepartition.Toseeallthecommandline
options available, use the man command to display the manual pages for the filesystem
command (see Chapter 3). All the filesystem commands allow you to create a default
filesystemwithjustthesimplecommandwithnooptions:
$sudomkfs.ext4/dev/sdb1
[sudo]passwordforChristine:
mke2fs1.41.12(17-May-2010)
Filesystemlabel=
OStype:Linux
Blocksize=4096(log=2)
Fragmentsize=4096(log=2)
Stride=0blocks,Stripewidth=0blocks
131648inodes,526120blocks
26306blocks(5.00%)reservedforthesuperuser
Firstdatablock=0
Maximumfilesystemblocks=541065216
17blockgroups
32768blockspergroup,32768fragmentspergroup
7744inodespergroup
Superblockbackupsstoredonblocks:
32768,98304,163840,229376,294912
Writinginodetables:done
Creatingjournal(16384blocks):done
Writingsuperblocksandfilesystemaccountinginformation:done
Thisfilesystemwillbeautomaticallycheckedevery23mountsor
180days,whichevercomesfirst.Usetune2fs-cor-itooverride.
$
The new filesystem uses the ext4 filesystem type, which is a journaling filesystem in
Linux.Noticethatpartofthecreationprocesswastocreatethenewjournal.
After you create the filesystem for a partition, the next step is to mount it on a virtual
directorymountpointsoyoucanstoredatainthenewfilesystem.Youcanmountthenew
filesystemanywhereinyourvirtualdirectorywhereyouneedtheextraspace.
$ls/mnt
$
$sudomkdir/mnt/my_partition
$
$ls-al/mnt/my_partition/
$
$ls-dF/mnt/my_partition
/mnt/my_partition/
$
$sudomount-text4/dev/sdb1/mnt/my_partition
$
$ls-al/mnt/my_partition/
total24
drwxr-xr-x.3rootroot4096Jun1109:53.
drwxr-xr-x.3rootroot4096Jun1109:58..
drwx––.2rootroot16384Jun1109:53lost+found
$
The mkdircommand(Chapter3)createsthemountpointinthevirtualdirectory,andthe
mountcommandaddsthenewharddrivepartitiontothemountpoint.The -toptionon
themountcommandindicateswhatfilesystemtype,ext4,youaremounting.Nowyoucan
savenewfilesandfoldersonthenewpartition!
Note
Thismethodofmountingafilesystemonlytemporarilymountsthefilesystem.When
yourebootyourLinuxsystem,thefilesystemdoesn’tautomaticallymount.Toforce
Linuxtoautomaticallymountthenewfilesystematboottime,addthenew
filesystemtothe/etc/fstabfile.
Nowthatthefilesystemismountedwithinthevirtualdirectorysystem,itcanstarttobe
used on a regular basis. Unfortunately, with regular use comes the potential for serious
problems,suchasfilesystemcorruption.Thenextsectionlooksathowtodealwiththese
issues.
Checkingandrepairingafilesystem
Evenwithmodernfilesystems,thingscangowrongifpowerisunexpectedlylost,orifa
wayward application locks up the system while file access is in progress. Fortunately,
some command line tools are available to help you make an attempt to restore the
filesystembacktoorder.
Eachfilesystemhasitsownrecoverycommandforinteractingwiththefilesystem.That
hasthepotentialofgettingugly,becausemoreandmorefilesystemsareavailableinthe
Linux environment, making for lots of individual commands you have to know.
Fortunately, a common front-end program available can determine the filesystem on the
storage device and use the appropriate filesystem recovery command based on the
filesystembeingrecovered.
The fsck command is used to check and repair most Linux filesystem types, including
onesdiscussedearlierinthischapter—ext,ext2,ext3,ext4,Reiser4,JFS,andXFS.The
formatofthecommandis:
fsckoptionsfilesystem
Youcanlistmultiple filesystementriesonthecommandlinetocheck.Filesystemscan
bereferencedusingeitherthedevicename,themountpointinthevirtualdirectory,ora
specialLinuxUUIDvalueassignedtothefilesystem.
Tip
Althoughjournalingfilesystemsusersdoneedthefsckcommand,itisarguableasto
whetherCOWfilesystemsusersdo.Infact,theZFSfilesystemdoesnotevenhavean
interfacetothefsckutility
Thefsckcommandusesthe/etc/fstabfiletoautomaticallydeterminethefilesystemon
a storage device that’s normally mounted on the system. If the storage device isn’t
normallymounted(suchasifyoujustcreatedafilesystemonanewstoragedevice),you
needtousethe -tcommandlineoptiontospecifythefilesystemtype.Table8.4liststhe
othercommandlineoptionsavailable.
Table8.4ThefsckCommandLineOptions
Option
-a
-A
-C
-N
-r
-R
-s
-t
-T
-V
-y
Description
Automaticallyrepairsthefilesystemiferrorsaredetected
Checksallthefilesystemslistedinthe/etc/fstabfile
Displaysaprogressbarforfilesystemsthatsupportthatfeature(onlyext2and
ext3)
Doesn’trunthecheck,onlydisplayswhatcheckswouldbeperformed
Promptstofixiferrorsfound
Skipstherootfilesystemifusingthe-Aoption
Ifcheckingmultiplefilesystems,performsthechecksoneatatime
Specifiesthefilesystemtypetocheck
Doesn’tshowtheheaderinformationwhenstarting
Producesverboseoutputduringthechecks
Automaticallyrepairsthefilesystemiferrorsdetected
Youmaynoticethatsomeofthecommandlineoptionsareredundant.That’spartofthe
problemoftryingtoimplementacommonfront-endformultiplecommands.Someofthe
individual filesystem repair commands have additional options that can be used. If you
need to do more advanced error checking, you’ll need to check the man pages for the
individual filesystem repair tool to see if there are extended options specific to that
filesystem.
Tip
Youcanrunthefsckcommandonunmountedfilesystemsonly.Formost
filesystems,youcanjustunmountthefilesystemtocheckitandthenremountitwhen
you’refinished.However,becausetherootfilesystemcontainsallthecoreLinux
commandsandlogfiles,youcan’tunmountitonarunningsystem.
ThisisatimewherehavingaLinuxLiveCDcomesinhandy!Justbootyoursystem
withtheLiveCD,andthenrunthefsckcommandontherootfilesystem!
Thischapterhasshowedhowtohandlefilesystemscontainedinphysicalstoragedevices.
Linux also provides a couple of different ways to create logical storage devices for
filesystems.Thenextsectionexamineshowyoucanusealogicalstoragedeviceforyour
filesystems.
ManagingLogicalVolumes
If you create your filesystems using standard partitions on hard drives, trying to add
additionalspacetoanexistingfilesystemcanbesomewhatofapainfulexperience.You
canonlyexpandapartitiontotheextentoftheavailablespaceonthesamephysicalhard
drive.Ifnomorespaceisavailableonthatharddrive,you’restuckhavingtogetalarger
harddriveandmanuallymovingtheexistingfilesystemtothenewdrive.
What would come in handy is a way to dynamically add more space to an existing
filesystem by just adding a partition from another hard drive to the existing filesystem.
TheLinuxLogicalVolumeManager(LVM)softwarepackageallowsyoutodojustthat.It
providesaneasywayforyoutomanipulatediskspaceonaLinuxsystemwithouthaving
torebuildentirefilesystems.
Exploringlogicalvolumemanagementlayout
The core of logical volume management is how it handles the physical hard drive
partitions installed on the system. In the logical volume management world, hard drives
arecalledphysicalvolumes(PV).EachPVmapstoaspecificphysicalpartitioncreatedon
aharddrive.
Multiple PV elements are pooled together to create a volume group (VG). The logical
volumemanagementsystemtreatstheVGlikeaphysicalharddrive,butinrealitytheVG
may consist of multiple physical partitions spread across multiple hard drives. The VG
providesaplatformtocreatethelogicalpartitions,whichactuallycontainthefilesystem.
The final layer in the structure is the logicalvolume (LV). The LV creates the partition
environment for Linux to create a filesystem, acting similar to a physical hard disk
partitionasfarasLinuxisconcerned.TheLinuxsystemtreatstheLVjustlikeaphysical
partition.YoucanformattheLVusinganyoneofthestandardLinuxfilesystemsandthen
addittotheLinuxvirtualdirectoryatamountpoint.
Figure 8.1 shows the basic layout of a typical Linux logical volume management
environment.
Figure8.1Thelogicalvolumemanagementenvironment
Thevolumegroup,showninFigure8.1,spansacrossthreeseparatephysicalharddrives,
whichcontainfiveseparatephysicalpartitions.Insidethevolumegrouparetwoseparate
logical volumes. The Linux system treats each logical volume just like a physical
partition.Eachlogicalvolumecanbeformattedasanext4filesystemandthenmountedto
aspecificlocationinthevirtualdirectory.
Notice in Figure 8.1 that the third physical hard drive has an unused partition. Using
logical volume management, you can easily assign this unused partition to the existing
volumegroupatalatertime,andtheneitheruseittocreateanewlogicalvolumeoraddit
toexpandoneoftheexistinglogicalvolumeswhenyouneedmorespace.
Likewise,ifyouaddanewharddrivetothesystem,thelocalvolumemanagementsystem
allowsyoutoaddittotheexistingvolumegroup,andthencreatemorespaceforoneof
theexistinglogicalvolumes,orstartanewlogicalvolumetobemounted.That’samuch
betterwayofhandlingexpandingfilesystems!
UsingtheLVMinLinux
The Linux LVM was developed by Heinz Mauelshagen and released to the Linux
community in 1998. It allows you to manage a complete logical volume management
environmentinLinuxusingsimplecommandlinecommands.
TwoversionsofLinuxLVMareavailable:
LVM1:TheoriginalLVMpackagereleasedin1998andavailableinonlythe2.4
Linuxkernels.Itprovidesonlybasiclogicalvolumemanagementfeatures.
LVM2:AnupdatedversionoftheLVM,availableinthe2.6Linuxkernels.It
providesadditionalfeaturesoverthestandardLVM1features.
MostmodernLinuxdistributionsusingthe2.6kernelversionoraboveprovidesupportfor
LVM2.Besidesthestandardlogicalvolumemanagementfeatures,LVM2providesafew
othernicethingsforyoutouseinyourLinuxsystem.
TakingaSnapshot
TheoriginalLinuxLVMallowsyoutocopyanexistinglogicalvolumetoanotherdevice
whilethelogicalvolumeisactive.Thisfeatureiscalledasnapshot.Snapshotsaregreat
forbackingupimportantdatathatcan’tbelockedduetohighavailabilityrequirements.
Traditional backup methods usually lock files as they’re being copied to the backup
media. The snapshot allows you to continue running mission critical web or database
serverswhileperformingthecopy.Unfortunately,LVM1allowsyoutocreateonlyareadonlysnapshot.Afteryoucreatethesnapshot,youcan’twritetoit.
LVM2 allows you to create a read-write snapshot of an active logical volume. With the
read-writecopy,youcanremovetheoriginallogicalvolumeandmountthesnapshotasa
replacement.Thisfeatureisgreatforfastfail-oversorforexperimentingwithapplications
thatmodifydatathatmayneedtoberestoredifsomethingfails.
Striping
AnotherinterestingfeaturethatLVM2providesisstriping.Withstriping,alogicalvolume
iscreatedacrossmultiplephysicalharddrives.WhentheLinuxLVMwritesafiletothe
logicalvolume,thedatablocksinthefilearespreadacrossthemultipleharddrives.Each
successiveblockofdataiswrittentothenextharddrive.
Striping helps improve disk performance, because Linux can write the multiple data
blocksforafiletothemultipleharddrivessimultaneously,ratherthanhavingtowaitfora
single hard drive to move the read/write head to different locations. This improvement
also applies to reading sequentially accessed files, because the LVM can read data from
themultipleharddrivessimultaneously.
Note
LVMstripingisnotthesameasRAIDstriping.LVMstripingdoesn’tprovidea
parityentry,whichcreatesthefault-tolerantenvironment.Infact,LVMstripingmay
increasethechanceofafilebeinglostduetoaharddrivefailure.Asingledisk
failurecanresultinmultiplelogicalvolumesbeinginaccessible.
Mirroring
Just because you install a filesystem using LVM doesn’t mean that things can’t still go
wrong in the filesystem. Just as in a physical partition, LVM logical volumes are
susceptibletopoweroutagesanddiskcrashes.Afterafilesystembecomescorrupt,there’s
alwaysapossibilitythatyouwon’tbeabletorecoverit.
TheLVMsnapshotprocessprovidessomecomfortknowingthatyoucancreateabackup
copyofalogicalvolumeatanytime,butforsomeenvironmentsthatmaynotbeenough.
Systemsthathavelotsofdatachanges,suchasdatabaseservers,maystorehundredsor
thousandsofrecordssincethelastsnapshot.
A solution to this problem is the LVM mirror. A mirror is a complete copy of a logical
volume that’s updated in real time. When you create the mirror logical volume, LVM
synchronizestheoriginallogicalvolumetothemirrorcopy.Dependingonthesizeofthe
originallogicalvolume,thismaytakesometimetocomplete.
Aftertheoriginalsynchronizationiscomplete,LVMperformstwowritesforeachwrite
processinthefilesystem—onetothemainlogicalvolumeandonetothemirroredcopy.
As you can guess, this process does slow down write performance on the system.
However,iftheoriginallogicalvolumeshouldbecomecorruptforsomereason,youhave
acompleteup-to-datecopyatyourfingertips!
UsingtheLinuxLVM
Now that you’ve seen what the Linux LVM can do, this section discusses how to
implement it to help organize the disk space on your system. The Linux LVM package
only provides command line programs for creating and managing all the components in
thelogicalvolumemanagementsystem.SomeLinuxdistributionsincludegraphicalfrontendstothecommandlinecommands,butforcompletecontrolofyourLVMenvironment,
it’sbesttogetcomfortableworkingdirectlywiththecommands.
DefiningPhysicalVolumes
The first step in the process is to convert the physical partitions on the hard drive into
physicalvolumeextentsusedbytheLinuxLVM.Ourfriendthefdiskcommandhelpsus
here.AftercreatingthebasicLinuxpartition,youneedtochangethepartitiontypeusing
thetcommand:
[…]
Command(mforhelp):t
Selectedpartition1
Hexcode(typeLtolistcodes):8e
Changedsystemtypeofpartition1to8e(LinuxLVM)
Command(mforhelp):p
Disk/dev/sdb:5368MB,5368709120bytes
255heads,63sectors/track,652cylinders
Units=cylindersof16065*512=8225280bytes
Sectorsize(logical/physical):512bytes/512bytes
I/Osize(minimum/optimal):512bytes/512bytes
Diskidentifier:0xa8661341
DeviceBootStartEndBlocksIdSystem
/dev/sdb112622104483+8eLinuxLVM
Command(mforhelp):w
Thepartitiontablehasbeenaltered!
Callingioctl()tore-readpartitiontable.
Syncingdisks.
$
The8epartitiontypedenotesthatthepartitionwillbeusedaspartofaLinuxLVMsystem
andnotasadirectfilesystem,asyousawwiththe83partitiontypeearlier.
Note
Ifthepvcreatecommandinthenextstepdoesnotworkforyou,it’smostlikelydue
totheLVM2packagenotbeinginstalledbydefault.Toinstallthepackage,usethe
packagenamelvm2andseeChapter9forhowtoinstallsoftwarepackages.
Thenextstepistousethepartitiontocreatetheactualphysicalvolume.That’sdoneusing
the pvcreatecommand.The pvcreatecommanddefinesthephysicalpartitiontousefor
thePV.ItsimplytagsthepartitionasaphysicalvolumeintheLinuxLVMsystem:
$sudopvcreate/dev/sdb1
dev_is_mpath:failedtogetdevicefor8:17
Physicalvolume“/dev/sdb1”successfullycreated
$
Note
Don’tletthedauntingmessagedev_is_mpath:failedtogetdevicefor8:17
orsimilarmessagesfrightenyou.Aslongasyoureceivethesuccessfullycreated
message,alliswell.Thepvcreatecommandcheckstoseewhetherthepartitionisa
multi-path(mpath)device.Ifitisnot,itissuesthedauntingmessage.
Youcanusethepvdisplaycommandtodisplayalistofphysicalvolumesyou’vecreated
ifyou’dliketoseeyourprogressalongtheway:
$sudopvdisplay/dev/sdb1
“/dev/sdb1”isanewphysicalvolumeof“2.01GiB”
–NEWPhysicalvolume–
PVName/dev/sdb1
VGName
PVSize2.01GiB
AllocatableNO
PESize0
TotalPE0
FreePE0
AllocatedPE0
PVUUID0FIuq2-LBod-IOWt-8VeN-tglm-Q2ik-rGU2w7
$
The pvdisplaycommandshowsthat /dev/sdb1isnowtaggedasaPV.Notice,however,
thatintheoutput,theVGNameisblank.ThePVdoesnotyetbelongtoavolumegroup.
CreatingVolumeGroups
The next step in the process is to create one or more volume groups from the physical
volumes.Therearenosetrulesforhowmanyvolumegroupsyouneedtocreateforyour
system—youcanaddalltheavailablephysicalvolumestoasinglevolumegroup,oryou
cancreatemultiplevolumegroupsbycombiningdifferentphysicalvolumes.
To create the volume group from the command line, you need to use the vgcreate
command.Thevgcreatecommandrequiresafewcommandlineparameterstodefinethe
volumegroupname,aswellasthenameofthephysicalvolumesyou’reusingtocreate
thevolumegroup:
$sudovgcreateVol1/dev/sdb1
Volumegroup“Vol1”successfullycreated
$
That’s not all too exciting for output! If you’d like to see some details about the newly
createdvolumegroup,usethevgdisplaycommand:
$sudovgdisplayVol1
–Volumegroup–
VGNameVol1
SystemID
Formatlvm2
MetadataAreas1
MetadataSequenceNo1
VGAccessread/write
VGStatusresizable
MAXLV0
CurLV0
OpenLV0
MaxPV0
CurPV1
ActPV1
VGSize2.00GiB
PESize4.00MiB
TotalPE513
AllocPE/Size0/0
FreePE/Size513/2.00GiB
VGUUIDoe4I7e-5RA9-G9ti-ANoI-QKLz-qkX4-58Wj6e
$
Thisexamplecreatesavolumegroupnamed Vol1,usingthephysicalvolumecreatedon
the/dev/sdb1partition.
Nowthatyouhaveoneormorevolumegroupscreated,you’rereadytocreatethelogical
volume.
CreatingLogicalVolumes
ThelogicalvolumeiswhattheLinuxsystemusestoemulateaphysicalpartition,andit
holds the filesystem. The Linux system handles the logical volumes just like a physical
partition, allowing you to define filesystems in the logical volume and then mount the
filesystemintothevirtualdirectory.
Tocreatethelogicalvolume,usethe lvcreatecommand.Althoughyoucanusuallyget
away without using command line options in the other Linux LVM commands, the
lvcreate command requires at least some options to be entered. Table 8.5 shows the
availablecommandlineoptions.
Table8.5ThelvcreateOptions
Option
Long
Option
Name
Description
-c
—chunksize
Specifiesthechunksizeofthesnapshotlogicalvolume
-C
—
contiguous
Setsorresetsthecontiguousallocationpolicy
-i
-I
-l
-L
-m
Specifiesthenumberofstripes
—stripsize
Specifiesthesizeofeachstripe
Specifiesthenumberoflogicalextentstoallocatetoanewlogical
—extents
volumeorthepercentofthelogicalextentstouse
—size
Specifiesthedisksizetoallocatetoanewlogicalvolume
—minor
Specifiestheminornumberofthedevice
—mirrors
Createsamirroredlogicalvolume
—stripes
-M
—
persistent
Makestheminornumberpersistent
-n
—name
Specifiesthenameofthenewlogicalvolume
-p
—
permission
Setsread/writepermissionforthelogicalvolume
-r
—readahead
Setsthereadaheadsectorcount
-R
—
regionsize
Specifiesthesizetodividethemirrorregionsinto
-s
—snapshot
-Z
—zero
Createsasnapshotlogicalvolume
Setsthefirst1KBofdataonthenewlogicalvolumetozeros
Although the command line options may look intimidating, for most situations, you can
getbywithaminimalamountofoptions:
$sudolvcreate-l100%FREE-nlvtestVol1
Logicalvolume“lvtest”created
$
Ifyouwanttoseethedetailsofwhatyoucreated,usethelvdisplaycommand:
$sudolvdisplayVol1
–Logicalvolume–
LVPath/dev/Vol1/lvtest
LVNamelvtest
VGNameVol1
LVUUID4W2369-pLXy-jWmb-lIFN-SMNX-xZnN-3KN208
LVWriteAccessread/write
LVCreationhost,time…-0400
LVStatusavailable
#open0
LVSize2.00GiB
CurrentLE513
Segments1
Allocationinherit
Readaheadsectorsauto
-currentlysetto256
Blockdevice253:2
$
Nowyoucanseejustwhatyoucreated!Noticethatthevolumegroupname(Vol1)isused
toidentifythevolumegrouptousewhencreatingthenewlogicalvolume.
The-lparameterdefineshowmuchoftheavailablespaceonthevolumegroupspecified
touseforthelogicalvolume.Noticethatyoucanspecifythevalueasapercentofthefree
space in the volume group. This example used all (100%) of the free space for the new
logicalvolume.
Youcanusethe -lparametertospecifythesizeasapercentageoftheavailablespaceor
the -Lparametertospecifytheactualsizeinbytes,kilobytes(KB),megabytes(MB),or
gigabytes(GB).The -n parameter allows you to provide a name for the logical volume
(calledlvtestinthisexample).
CreatingtheFilesystem
After you run the lvcreate command, the logical volume exists but doesn’t have a
filesystem. To do that, you need to use the appropriate command line program for the
filesystemyouwanttocreate:
$sudomkfs.ext4/dev/Vol1/lvtest
mke2fs1.41.12(17-May-2010)
Filesystemlabel=
OStype:Linux
Blocksize=4096(log=2)
Fragmentsize=4096(log=2)
Stride=0blocks,Stripewidth=0blocks
131376inodes,525312blocks
26265blocks(5.00%)reservedforthesuperuser
Firstdatablock=0
Maximumfilesystemblocks=541065216
17blockgroups
32768blockspergroup,32768fragmentspergroup
7728inodespergroup
Superblockbackupsstoredonblocks:
32768,98304,163840,229376,294912
Writinginodetables:done
Creatingjournal(16384blocks):done
Writingsuperblocksandfilesystemaccountinginformation:done
Thisfilesystemwillbeautomaticallycheckedevery28mountsor
180days,whichevercomesfirst.Usetune2fs-cor-itooverride.
$
After you’ve created the new filesystem, you can mount the volume in the virtual
directoryusingthestandardLinuxmountcommand,justasifitwereaphysicalpartition.
Theonlydifferenceisthatyouuseaspecialpaththatidentifiesthelogicalvolume:
$sudomount/dev/Vol1/lvtest/mnt/my_partition
$
$mount
/dev/mapper/vg_server01-lv_rooton/typeext4(rw)
[…]
/dev/mapper/Vol1-lvteston/mnt/my_partitiontypeext4(rw)
$
$cd/mnt/my_partition
$
$ls-al
total24
drwxr-xr-x.3rootroot4096Jun1210:22.
drwxr-xr-x.3rootroot4096Jun1109:58..
drwx––.2rootroot16384Jun1210:22lost+found
$
Notice that the path used in both the mkfs.ext4 and mount commands is a little odd.
Insteadofaphysicalpartitionpath,thepathusesthevolumegroupname,alongwiththe
logicalvolumename.Afterthefilesystemismounted,youcanaccessthenewareainthe
virtualdirectory.
ModifyingtheLVM
BecausethebenefitofusingtheLinuxLVMistodynamicallymodifyfilesystems,you’d
expectthatsometoolswouldallowyoutodothat.SometoolsareavailableinLinuxthat
allowyoutomodifytheexistinglogicalvolumemanagementconfiguration.
If you don’t have access to a fancy graphical interface for managing your Linux LVM
environment,allisnotlost.You’vealreadyseensomeoftheLinuxLVMcommandline
programsinactioninthischapter.Youcanuseahostofothercommandlineprogramsto
manage the LVM setup after you’ve installed it. Table8.6 lists the common commands
thatareavailableintheLinuxLVMpackage.
Table8.6TheLinuxLVMCommands
Command
vgchange
vgremove
vgextend
vgreduce
lvextend
lvreduce
Function
Activatesanddeactivatesavolumegroup
Removesavolumegroup
Addsphysicalvolumestoavolumegroup
Removesphysicalvolumesfromavolumegroup
Increasesthesizeofalogicalvolume
Decreasesthesizeofalogicalvolume
Using these command line programs, you have full control over your Linux LVM
environment.
Tip
Becarefulwhenmanuallyincreasingordecreasingthesizeofalogicalvolume.The
filesystemstoredinthelogicalvolumemustbemanuallyfixedtohandlethechange
insize.Mostfilesystemsincludecommandlineprogramsforreformattingthe
filesystem,suchastheresize2fsprogramfortheext2,ext3,andext4filesystems.
Summary
Working with storage devices in Linux requires that you know a little bit about
filesystems. Knowing how to create and work with filesystems from the command line
cancomeinhandyasyouworkonLinuxsystems.Thischapterdiscussedhowtohandle
filesystemsfromtheLinuxcommandline.
TheLinuxsystemisdifferentfromWindowsinthatitsupportslotsofdifferentmethods
for storing files and folders. Each filesystem method has different features that make it
ideal for different situations. Also, each filesystem method uses different commands for
interactingwiththestoragedevice.
Beforeyoucaninstallafilesystemonastoragedevice,youmustfirstpreparethedevice.
The fdisk command is used to partition storage devices to get them ready for the
filesystem. When you partition the storage device, you must define what type of
filesystemwillbeusedonit.
Afteryoupartitionastoragedevice,youcanuseoneofseveraldifferentfilesystemsfor
thepartition.PopularLinuxfilesystemsincludeext4andXFS.Bothofthesefilesystems
providejournalingfilesystemfeatures,makingthemlesspronetoerrorsandproblemsif
theLinuxsystemshouldcrash.
Onelimitingfactortocreatingfilesystemsdirectlyonastoragedevicepartitionisthatyou
can’teasilychangethesizeofthefilesystemifyourunoutofdiskspace.However,Linux
supports logical volume management, a method of creating virtual partitions across
multiplestoragedevices.Thismethodallowsyoutoeasilyexpandanexistingfilesystem
withouthavingtocompletelyrebuildit.TheLinuxLVMpackageprovidescommandline
commands to create logical volumes across multiple storage devices on which to build
filesystems.
Nowthatyou’veseenthecoreLinuxcommandlinecommands,it’sclosetothetimeto
start creating some shell script programs. However, before you start coding, we need to
discussanotherelement:installingsoftware.Ifyouplantowriteshellscripts,youneedan
environment in which to create your masterpieces. The next chapter discusses how to
install and manage software packages from the command line in different Linux
environments.
Chapter9
InstallingSoftware
InThisChapter
1. Installingsoftware
2. UsingDebianpackages
3. WorkingwithRedHatpackages
IntheolddaysofLinux,installingsoftwarecouldbeapainfulexperience.
Fortunately,theLinuxdevelopershavemadelifealittleeasierforusbybundling
softwareintopre-builtpackagesthataremucheasiertoinstall.However,youstill
havealittleworktodotogetthesoftwarepackagesinstalled,especiallyifyouwant
todothatfromthecommandline.ThischapterlooksatthevariousPackage
ManagementSystemsavailableinLinuxandthecommandlinetoolsusedfor
softwareinstallation,management,andremoval.
PackageManagementPrimer
Before diving into the world of Linux software package management, this chapter goes
throughafewofthebasicsfirst.EachofthemajorLinuxdistributionsutilizessomeform
ofaPackageManagementSystem(PMS)tocontrolinstallingsoftwareapplicationsand
libraries.APMSutilizesadatabasethatkeepstrackoftheseitems:
WhatsoftwarepackagesareinstalledontheLinuxsystem
Whatfileshavebeeninstalledforeachpackage
Versionsofeachofthesoftwarepackagesinstalled
Softwarepackagesarestoredonservers,calledrepositories,andareaccessedacrossthe
Internet via PMS utilities running on your local Linux system. You can use the PMS
utilitiestosearchfornewsoftwarepackagesorevenupdatestosoftwarepackagesalready
installedonthesystem.
Asoftwarepackageoftenhasdependenciesorotherpackagesthatmustbeinstalledfirst
forthesoftwaretorunproperly.ThePMSutilitiesdetectthesedependenciesandofferto
installanyadditionallyneededsoftwarepackagesbeforeinstallingthedesiredpackage.
The downside to PMS is that there isn’t a single standard utility. Whereas all the bash
shellcommandsdiscussedsofarinthisbookworknomatterwhichLinuxdistributionyou
use,thisisnottruewithsoftwarepackagemanagement.
ThePMSutilitiesandtheirassociatedcommandsarevastlydifferentbetweenthevarious
Linux distributions. The two primary PMS base utilities commonly used in the Linux
worldaredpkgandrpm.
Debian-baseddistributionssuchasUbuntuandLinuxMintuse,atthebaseoftheirPMS
utilities,the dpkgcommand.ThiscommandinteractsdirectlywiththePMSontheLinux
systemandisusedforinstalling,managing,andremovingsoftwarepackages.
TheRedHat–baseddistributions,suchasFedora,openSUSE,andMandriva,usethe rpm
commandatthebaseoftheirPMS.Similartothe dpkgcommand,the rpmcommandcan
listinstalledpackages,installnewpackages,andremoveexistingsoftware.
NotethatthesetwocommandsarethecoreoftheirrespectivePMS,nottheentirePMS
itself. Many Linux distributions that use the dpkg or rpm methods have built additional
specialtyPMSutilitiesuponthesebasecommandstohelpmakeyourlifemucheasier.The
following sections walk through various PMS utility commands you’ll run into in the
popularLinuxdistributions.
TheDebian-BasedSystems
The dpkgcommandisatthecoreoftheDebian-basedfamilyofPMStools.Theseother
toolsareincludedinthisPMS:
apt-get
apt-cache
aptitude
Byfarthemostcommoncommandlinetoolisaptitude,andforgoodreason.Theaptitude
toolisessentiallyafront-endforboththeapttoolsanddpkg.WhereasdpkgisaPMStool,
aptitudeisacompletePackageManagementSystem.
Using the aptitude command at the command line helps you avoid common software
installation problems, such as missing software dependencies, unstable system
environments,andjustawholelotofunnecessaryhassle.Thissectionlooksathowtouse
theaptitudecommandtoolfromtheLinuxcommandline.
Managingpackageswithaptitude
AcommontaskfacedbyLinuxsystemadministratorsistodeterminewhatpackagesare
alreadyinstalledonthesystem.Fortunately,aptitudehasahandyinteractiveinterfacethat
makesthistaskaneasyone.
If you have aptitude installed in your Linux distribution, at the shell prompt just type
aptitudeandpressEnter.Youarethrownintoaptitude’sfull-screenmode,asyoucansee
inFigure9.1.
Figure9.1Theaptitudemainwindow
Use the arrow keys to maneuver around the menu. Select the menu option Installed
Packages to see what packages are installed. You will see several groups of software
packages,suchaseditors,andsoon.Anumberinparenthesesfollowseachgroup,which
indicatesthenumberofpackagesthegroupcontains.
Use the arrow keys to highlight a group, and press Enter to see each subgroup of
packages. You then see the individual package names and their version numbers. Press
Enter on individual packages to get very detailed information, such as the package’s
description,homepage,size,maintainer,andsoon.
Whenyou’refinishedviewingtheinstalledpackages,press qtoquitthedisplay.Youcan
thengobacktothearrowkeys.anduseEntertotoggleopenorclosedthepackagesand
theirsubgroups.Whenyouareallfinished,justpress qmultipletimesuntilyoureceive
thepop-upscreen“ReallyquitAptitude?”
If you already know the packages on your system and want to quickly display detailed
information about a particular package, you don’t need to go into aptitude’s interactive
interface.Youcanuseaptitudeasasinglecommandatthecommandline:
aptitudeshowpackage_name
Here’sanexampleofdisplayingthedetailsofthepackagemysql-client:
$aptitudeshowmysql-client
Package:mysql-client
State:notinstalled
Version:5.5.38-0ubuntu0.14.04.1
Priority:optional
Section:database
Maintainer:UbuntuDevelopers<[email protected]>
Architecture:all
UncompressedSize:129k
Depends:mysql-client-5.5
Providedby:mysql-client-5.5
Description:MySQLdatabaseclient(metapackagedependingonthelatest
version)
Thisisanemptypackagethatdependsonthecurrent”best”versionof
mysql-client(currentlymysql-client-5.5),asdeterminedbytheMySQL
maintainers.InstallthispackageifindoubtaboutwhichMySQLversion
you
want,asthisistheoneconsideredtobeinthebestshapebythe
Maintainers.
Homepage:http://dev.mysql.com/
$
Note
Theaptitudeshowcommandindicatesthatthepackageisnotinstalledonthe
system.Italsoshowsdetailedpackageinformationfromthesoftwarerepository.
One detail you cannot get with aptitude is a listing of all the files associated with a
particularsoftwarepackage.Togetthislist,youmustgotothedpkgtoolitself:
dpkg-Lpackage_name
Here’s an example of using dpkg to list all the files installed as part of the vim-common
package:
$
$dpkg-Lvim-common
/.
/usr
/usr/bin
/usr/bin/xxd
/usr/bin/helpztags
/usr/lib
/usr/lib/mime
/usr/lib/mime/packages
/usr/lib/mime/packages/vim-common
/usr/share
/usr/share/man
/usr/share/man/ru
/usr/share/man/ru/man1
/usr/share/man/ru/man1/vim.1.gz
/usr/share/man/ru/man1/vimdiff.1.gz
/usr/share/man/ru/man1/xxd.1.gz
/usr/share/man/it
/usr/share/man/it/man1
[…]
$
Youcanalsodothereverse—findwhatpackageaparticularfilebelongsto:
dpkg—searchabsolute_file_name
Notethatyouneedtouseanabsolutefilereferenceforthistowork:
$
$dpkg—search/usr/bin/xxd
vim-common:/usr/bin/xxd
$
The output shows the /usr/bin/xxd file was installed as part of the vim-common
package.
Installingsoftwarepackageswithaptitude
Nowthatyouknowmoreaboutlistingsoftwarepackageinformationonyoursystem,this
section walks through a software package install. First, you’ll want to determine the
package name to install. How do you find a particular software package? Use the
aptitudecommandwiththesearchoption:
aptitudesearchpackage_name
The beauty of the search option is that you do not need to insert wildcards around
package_name.Wildcardsareimplied.Here’sanexampleofusing aptitudetolookfor
thewinesoftwarepackage:
$
$aptitudesearchwine
pgnome-wine-icon-theme-redvariationoftheGNOME-…
vlibkwineffects1-apiplibkwineffects1a-libraryusedbyeffects…
pq4wine-Qt4GUIforwine(W.I.N.E)
pshiki-wine-theme-redvariationoftheShiki-…
pwine-MicrosoftWindowsCompatibility…
pwine-dev-MicrosoftWindowsCompatibility…
pwine-gecko-MicrosoftWindowsCompatibility…
pwine1.0-MicrosoftWindowsCompatibility…
pwine1.0-dev-MicrosoftWindowsCompatibility…
pwine1.0-gecko-MicrosoftWindowsCompatibility…
pwine1.2-MicrosoftWindowsCompatibility…
pwine1.2-dbg-MicrosoftWindowsCompatibility…
pwine1.2-dev-MicrosoftWindowsCompatibility…
pwine1.2-gecko-MicrosoftWindowsCompatibility…
pwinefish-LaTeXEditorbasedonBluefish
$
Noticethatbeforeeachpackagenameiseithera pori.Ifyouseean iu,thepackageis
currentlyinstalledonyoursystem.Ifyouseea porv,itisavailablebutnotinstalled.As
youcanseefromtheprecedinglisting,thissystemdoesnothavewinecurrentlyinstalled,
butthepackageisavailablefromthesoftwarerepository.
Installingasoftwarepackageonasystemfromarepositoryusing aptitudeisaseasyas
this:
aptitudeinstallpackage_name
After you find the software package name from the search option, just plug it into the
aptitudecommandusingtheinstalloption:
$
$sudoaptitudeinstallwine
ThefollowingNEWpackageswillbeinstalled:
cabextract{a}esound-clients{a}esound-common{a}gnome-exe-thumbnailer
{a}
icoutils{a}imagemagick{a}libaudio2{a}libaudiofile0{a}libcdt4{a}
libesd0{a}libgraph4{a}libgvc5{a}libilmbase6{a}libmagickcore3-extra
{a}
libmpg123-0{a}libnetpbm10{a}libopenal1{a}libopenexr6{a}
libpathplan4{a}libxdot4{a}netpbm{a}ttf-mscorefonts-installer{a}
ttf-symbol-replacement{a}winbind{a}winewine1.2{a}wine1.2-gecko{a}
0packagesupgraded,27newlyinstalled,0toremoveand0notupgraded.
Needtoget0B/27.6MBofarchives.Afterunpacking121MBwillbeused.
Doyouwanttocontinue?[Y/n/?]Y
Preconfiguringpackages…
[…]
Alldone,noerrors.
Allfontsdownloadedandinstalled.
Updatingfontconfigcachefor/usr/share/fonts/truetype/msttcorefonts
Settingupwinbind(2:3.5.4~dfsg-1ubuntu7)…
*StartingtheWinbinddaemonwinbind
[OK]
Settingupwine(1.2-0ubuntu5)…
Settingupgnome-exe-thumbnailer(0.6-0ubuntu1)…
Processingtriggersforlibc-bin…
ldconfigdeferredprocessingnowtakingplace
$
Note
Beforetheaptitudecommandintheprecedinglisting,thesudocommandisused.
Thesudocommandallowsyoutorunacommandastherootuser.Youcanusethe
sudocommandtorunadministrativetasks,suchasinstallingsoftware.
Tocheckiftheinstallationprocessedproperly,justusethesearchoptionagain.Thistime
you should see an i u listed in front of the wine software package, indicating it is
installed.
Youmayalsonoticethatthereareadditionalpackageswiththe iuinfrontofthem.This
is because aptitude automatically resolved any necessary package dependencies for us
and installs the needed additional library and software packages. This is a wonderful
featureincludedinmanyPackageManagementSystems.
Updatingsoftwarewithaptitude
Whileaptitudehelpsprotectyoufromproblemsinstallingsoftware,tryingtocoordinate
a multiple-package update with dependencies can get tricky. To safely update all the
software packages on a system with any new versions in the repository, use the safeupgradeoption:
aptitudesafe-upgrade
Notice that this command doesn’t take a software package name as an argument. That’s
becausethe safe-upgrade option upgrades all the installed packages to the most recent
versionavailableintherepository,whichissaferforsystemstabilization.
Here’sasampleoutputfromrunningtheaptitudesafe-updatecommand:
$
$sudoaptitudesafe-upgrade
Thefollowingpackageswillbeupgraded:
evolutionevolution-commonevolution-pluginsgsfontslibevolution
xserver-xorg-video-geode
6packagesupgraded,0newlyinstalled,0toremoveand0notupgraded.
Needtoget9,312kBofarchives.Afterunpacking0Bwillbeused.
Doyouwanttocontinue?[Y/n/?]Y
Get:1http://us.archive.ubuntu.com/ubuntu/maverick/main
libevolutioni3862.30.3-1ubuntu4[2,096kB]
[…]
Preparingtoreplacexserver-xorg-video-geode2.11.9-2
(using…/xserver-xorg-video-geode_2.11.9-3_i386.deb)…
Unpackingreplacementxserver-xorg-video-geode…
Processingtriggersforman-db…
Processingtriggersfordesktop-file-utils…
Processingtriggersforpython-gmenu…
[…]
Currentstatus:0updates[-6].
$
Youcanalsouseless-conservativeoptionsforsoftwareupgrades:
aptitudefull-upgrade
aptitudedist-upgrade
These options perform the same task, upgrading all the software packages to the latest
versions. Where they differ from safe-upgrade is that they do not check dependencies
betweenpackages.Thewholepackagedependencyissuecangetrealugly.Ifyou’renot
exactly sure of the dependencies for the various packages, stick with the safe-upgrade
option.
Note
Obviously,runningaptitude’ssafe-upgradeoptionissomethingyoushoulddoona
regularbasistokeepyoursystemuptodate.However,itisespeciallyimportantto
runitafterafreshdistributioninstallation.Usually,lotsofsecuritypatchesand
updateshavebeenreleasedsincethelastfullreleaseofadistribution.
Uninstallingsoftwarewithaptitude
Gettingridofsoftwarepackageswithaptitudeisaseasyasinstallingandupgradingthem.
The only real choice you have to make is whether to keep the software’s data and
configurationfilesaroundafterward.
To remove a software package, but not the data and configuration files, use the remove
optionofaptitude.Toremoveasoftwarepackageandtherelateddataandconfiguration
files,usethepurgeoption:
$sudoaptitudepurgewine
[sudo]passwordforuser:
ThefollowingpackageswillbeREMOVED:
cabextract{u}esound-clients{u}esound-common{u}gnome-exe-thumbnailer
{u}
icoutils{u}imagemagick{u}libaudio2{u}libaudiofile0{u}libcdt4{u}
libesd0{u}libgraph4{u}libgvc5{u}libilmbase6{u}libmagickcore3-extra
{u}
libmpg123-0{u}libnetpbm10{u}libopenal1{u}libopenexr6{u}
libpathplan4{u}libxdot4{u}netpbm{u}ttf-mscorefonts-installer{u}
ttf-symbol-replacement{u}winbind{u}wine{p}wine1.2{u}wine1.2-gecko
{u}
0packagesupgraded,0newlyinstalled,27toremoveand6notupgraded.
Needtoget0Bofarchives.Afterunpacking121MBwillbefreed.
Doyouwanttocontinue?[Y/n/?]Y
(Readingdatabase…120968filesanddirectoriescurrentlyinstalled.)
Removingttf-mscorefonts-installer…
[…]
Processingtriggersforfontconfig…
Processingtriggersforureadahead…
Processingtriggersforpython-support…
$
Toseeifthepackagehasbeenremoved,youcanusethe aptitudesearchoptionagain.
Ifyouseea cinfrontofthepackagename,itmeansthesoftwarehasbeenremoved,but
the configuration files have not been purged from the system. A p in front indicates the
configurationfileshavealsobeenremoved.
Theaptituderepositories
Thedefaultsoftwarerepositorylocationsforaptitudearesetupforyouwhenyouinstall
your Linux distribution. The repository locations are stored in the file
/etc/apt/sources.list.
Inmanycases,youneverneedtoadd/removeasoftwarerepositorysoyoudon’tneedto
touchthisfile.However,aptitudepullssoftwarefromonlytheserepositories.Also,when
searchingforsoftwaretoinstallorupdate,aptitudechecksonlytheserepositories.Ifyou
needtoincludesomeadditionalsoftwarerepositoriesforyourPMS,thisistheplacetodo
it.
Tip
TheLinuxdistributiondevelopersworkhardtomakesurepackageversionsaddedto
therepositoriesdon’tconflictwithoneanother.Usuallyit’ssafesttoupgradeor
installasoftwarepackagefromtherepository.Evenifanewerversionisavailable
elsewhere,youmaywanttoholdoffinstallingituntilthatversionisavailableinyour
Linuxdistribution’srepository
Thefollowingisanexampleofasources.listfilefromanUbuntusystem:
$cat/etc/apt/sources.list
#debcdrom:[Ubuntu14.04LTS_TrustyTahr_-Releasei386(20140417)]/
trustymainrestricted
#Seehttp://help.ubuntu.com/community/UpgradeNotesforhowtoupgradeto
#newerversionsofthedistribution.
debhttp://us.archive.ubuntu.com/ubuntu/trustymainrestricted
deb-srchttp://us.archive.ubuntu.com/ubuntu/trustymainrestricted
##Majorbugfixupdatesproducedafterthefinalreleaseofthe
##distribution.
debhttp://us.archive.ubuntu.com/ubuntu/trusty-updatesmainrestricted
deb-srchttp://us.archive.ubuntu.com/ubuntu/trusty-updatesmainrestricted
##N.B.softwarefromthisrepositoryisENTIRELYUNSUPPORTEDbytheUbuntu
##team.Also,pleasenotethatsoftwareinuniverseWILLNOTreceiveany
##revieworupdatesfromtheUbuntusecurityteam.
debhttp://us.archive.ubuntu.com/ubuntu/trustyuniverse
deb-srchttp://us.archive.ubuntu.com/ubuntu/trustyuniverse
debhttp://us.archive.ubuntu.com/ubuntu/trusty-updatesuniverse
deb-srchttp://us.archive.ubuntu.com/ubuntu/trusty-updatesuniverse
[…]
##UncommentthefollowingtwolinestoaddsoftwarefromCanonical’s
##‘partner’repository.
##ThissoftwareisnotpartofUbuntu,butisofferedbyCanonicalandthe
##respectivevendorsasaservicetoUbuntuusers.
#debhttp://archive.canonical.com/ubuntutrustypartner
#deb-srchttp://archive.canonical.com/ubuntutrustypartner
##ThissoftwareisnotpartofUbuntu,butisofferedbythird-party
##developerswhowanttoshiptheirlatestsoftware.
debhttp://extras.ubuntu.com/ubuntutrustymain
deb-srchttp://extras.ubuntu.com/ubuntutrustymain
$
First,noticethatthefileisfullofhelpfulcommentsandwarnings.Therepositorysources
specifiedusethefollowingstructure:
deb(ordeb-src)addressdistribution_namepackage_type_list
The debor deb-srcvalueindicatesthesoftwarepackagetype.The debvalueindicatesit
is a source of compiled programs, whereas the deb-src value indicates it is a source of
sourcecode.
The address entry is the software repository’s web address. The distribution_name
entry is the name of this particular software repository’s distribution’s version. In the
example, the distribution name is trusty. This does not necessarily mean that the
distributionyouarerunningisUbuntu’sTrustyTahr;itjustmeanstheLinuxdistribution
is using the Ubuntu Trusty Tahr software repositories! For example, in Linux Mint’s
sources.listfile,youseeamixofLinuxMintandUbuntusoftwarerepositories.
Finally,thepackage_type_listentrymaybemorethanonewordandindicateswhattype
of packages the repository has in it. For example, you may see values such as main,
restricted,universe,orpartner.
Whenyouneedtoaddasoftwarerepositorytoyoursourcesfile,youcantrytowingit
yourself,butthatmorethanlikelywillcauseproblems.Often,softwarerepositorysitesor
various package developer sites have an exact line of text that you can copy from their
websiteandpasteintoyour sources.listfile.It’sbesttochoosethesaferrouteandjust
copy/paste.
Thefront-endinterface, aptitude,providesintelligentcommandlineoptionsforworking
with the Debian-based dpkg utility. Now it’s time to look at the Red Hat–based
distributions’rpmutilityanditsvariousfront-endinterfaces.
TheRedHat–BasedSystems
Like the Debian-based distributions, the Red Hat–based systems have several different
front-endtoolsthatareavailable.Thesearethecommonones:
yum:UsedinRedHatandFedora
urpm:UsedinMandriva
zypper:UsedinopenSUSE
These front-ends are all based on the rpm command line tool. The following section
discusses how to manage software packages using these various rpm-based tools. The
focusisonyum,butinformationisalsoincludedforzypperandurpm.
Listinginstalledpackages
To find out what is currently installed on your system, at the shell prompt, type the
followingcommand:
yumlistinstalled
Theinformationwillprobablywhizbyyouonthedisplayscreen,soit’sbesttoredirect
theinstalledsoftwarelistingintoafile.Youcanthenusethemoreorlesscommand(ora
GUIeditor)tolookatthelistinacontrolledmanner.
yumlistinstalled>installed_software
To list out the installed packages on your openSUSE or Mandriva distribution, see the
commandsinTable9.1.Unfortunately,the urpmtoolusedinMandrivacannotproducea
currentlyinstalledsoftwarelisting.Thus,youneedtoreverttotheunderlyingrpmtool.
Table9.1HowtoListInstalledSoftwarewithzypperandurpm
Distribution Front-EndTool
Command
urpm
rpm-qa>installed_software
Mandriva
zypper
zippersearch-I>installed_software
openSUSE
To find out detailed information for a particular software package, yum really shines. It
givesyouaveryverbosedescriptionofthepackage,andwithanothersimplecommand,
youcanseewhetherthepackageisinstalled:
#yumlistxterm
Loadedplugins:langpacks,presto,refresh-packagekit
Addingen_UStolanguagelist
AvailablePackages
xterm.i686253-1.el6
#
#yumlistinstalledxterm
Loadedplugins:refresh-packagekit
Error:NomatchingPackagestolist
#
Thecommandstolistdetailedsoftwarepackageinformationusingurpmandzypperarein
Table9.2. You can acquire an even more detailed set of package information from the
repository,usingtheinfooptiononthezyppercommand.
Table9.2HowtoSeeVariousPackageDetailswithzypperandurpm
DetailType
Package
Information
Installed?
Package
Information
Installed?
Front-End
Tool
Command
urpm
urpmq-ipackage_name
urpm
rpm-qpackage_name
zypper
zyppersearch-spackage_name
zypper
Samecommand,butlookforaniintheStatus
column
Finally,ifyouneedtofindoutwhatsoftwarepackageprovidesaparticularfileonyour
filesystem,theversatileyumcandothat,too!Justenterthecommand:
yumprovidesfile_name
Here’s an example of trying to find what software provided the configuration file
/etc/yum.conf:
#
#yumprovides/etc/yum.conf
Loadedplugins:fastestmirror,refresh-packagekit,security
Determiningfastestmirrors
*base:mirror.web-ster.com
*extras:centos.chi.host-engine.com
*updates:mirror.umd.edu
yum-3.2.29-40.el6.centos.noarch:RPMpackageinstaller/updater/manager
Repo:base
Matchedfrom:
Filename:/etc/yum.conf
yum-3.2.29-43.el6.centos.noarch:RPMpackageinstaller/updater/manager
Repo:updates
Matchedfrom:
Filename:/etc/yum.conf
yum-3.2.29-40.el6.centos.noarch:RPMpackageinstaller/updater/manager
Repo:installed
Matchedfrom:
Other:Provides-match:/etc/yum.conf
#
#
yum
checked three separate repositories: base, updates, and installed. From both, the
answeris:theyumsoftwarepackageprovidesthisfile!
Installingsoftwarewithyum
Installationofasoftwarepackageusing yumisincrediblyeasy.Thefollowingisthebasic
command for installing a software package, all its needed libraries, and package
dependenciesfromarepository:
yuminstallpackage_name
Here’sanexampleofinstallingthextermpackagethatwetalkedaboutinChapter2:
$suPassword:
#yuminstallxterm
Loadedplugins:fastestmirror,refresh-packagekit,security
Determiningfastestmirrors
*base:mirrors.bluehost.com
*extras:mirror.5ninesolutions.com
*updates:mirror.san.fastserv.com
SettingupInstallProcess
ResolvingDependencies
—>Runningtransactioncheck
–>Packagexterm.i6860:253-1.el6willbeinstalled
—>FinishedDependencyResolution
DependenciesResolved
[…]
Installed:
xterm.i6860:253-1.el6
Complete!
#
Note
Beforetheyumcommandintheprecedinglisting,thesu-commandisused.This
commandallowsyoutoswitchtotherootuser.OnthisLinuxsystem,the#denotes
youareloggedinasroot.Youshouldonlyswitchtorootusertemporarilyinorderto
runadministrativetasks,suchasinstallingandupdatingsoftware.Thesudo
commandisanotheroptionaswell.
Youcanalsomanuallydownloadan rpminstallationfileandinstallitusing yum.Thisis
calledalocalinstallation.Thisisthebasiccommand:
yumlocalinstallpackage_name.rpm
You can begin to see that one of yum‘s strengths is that it uses very logical and userfriendlycommands.
Table9.3showshowtoperformapackageinstallwith urpmandzypper.Youshouldnote
thatifyouarenotloggedinasroot,yougeta“commandnotfound”errormessageusing
urpm.
Table9.3HowtoInstallSoftwarewithzypperandurpm
Front-EndTool
Command
urpm
urpmipackage_name
zypper
zypperinstallpackage_name
Updatingsoftwarewithyum
In most Linux distributions, when you’re working away in the GUI, you get those nice
littlenotificationiconstellingyouthatanupdateisneeded.Hereatthecommandline,it
takesalittlemorework.
Toseethelistofalltheavailableupdatesforyourinstalledpackages,typethefollowing
command:
yumlistupdates
It’salwaysnicetogetnoresponsetothiscommandbecauseitmeansyouhavenothingto
update!However,ifyoudodiscoveraparticularsoftwarepackageneedsupdating,type
thefollowingcommand:
yumupdatepackage_name
If you’d like to update all the packages listed in the update list, just enter the following
command:
yumupdate
Commands for updating software packages on Mandriva and openSUSE are listed in
Table9.4.Whenurpmisused,therepositorydatabaseisautomaticallyrefreshedaswell
assoftwarepackagesupdated.
Table9.4HowtoUpdateSoftwarewithzypperandurpm
Front-EndTool
Command
urpm
urpmi—auto-update—update
zypper
zypperupdate
Uninstallingsoftwarewithyum
The yumtoolalsoprovidesaneasywaytouninstallsoftwareyounolongerwantonyour
system. As with aptitude, you need to choose whether to keep the software package’s
dataandconfigurationfiles.
To just remove the software package and keep any configuration and data files, use the
followingcommand:
yumremovepackage_name
Touninstallthesoftwareandallitsfiles,usetheeraseoption:
yumerasepackage_name
Itisequallyeasytoremovesoftwareusing urpm and zypperinTable9.5.Bothofthese
toolsperformafunctionsimilartoyum‘seraseoption.
Table9.5HowtoUninstallSoftwarewithzypperandurpm
Front-EndTool
Command
urpm
urpmepackage_name
zypper
zypperremovepackage_name
Although life is considerably easier with PMS packages, it’s not always problem-free.
Occasionally,thingsdogowrong.Fortunately,there’shelp.
Dealingwithbrokendependencies
Sometimes, as multiple software packages get loaded, a software dependency for one
packagecangetoverwrittenbytheinstallationofanotherpackage.Thisiscalledabroken
dependency.
Ifthisshouldhappenonyoursystem,firsttrythefollowingcommand:
yumcleanall
Thentrytousethe updateoptioninthe yumcommand.Sometimes,justcleaningupany
misplacedfilescanhelp.
Ifthatdoesn’tsolvetheproblem,trythefollowingcommand:
yumdeplistpackage_name
Thiscommanddisplaysallthepackage’slibrarydependenciesandwhatsoftwarepackage
providesthem.Afteryouknowthelibrariesrequiredforapackage,youcantheninstall
them.Here’sanexampleofdeterminingthedependenciesforthextermpackage:
#yumdeplistxterm
Loadedplugins:fastestmirror,refresh-packagekit,security
Loadingmirrorspeedsfromcachedhostfile
*base:mirrors.bluehost.com
*extras:mirror.5ninesolutions.com
*updates:mirror.san.fastserv.com
Findingdependencies:
package:xterm.i686253-1.el6
dependency:libncurses.so.5
provider:ncurses-libs.i6865.7-3.20090208.el6
dependency:libfontconfig.so.1
provider:fontconfig.i6862.8.0-3.el6
dependency:libXft.so.2
provider:libXft.i6862.3.1-2.el6
dependency:libXt.so.6
provider:libXt.i6861.1.3-1.el6
dependency:libX11.so.6
provider:libX11.i6861.5.0-4.el6
dependency:rtld(GNU_HASH)
provider:glibc.i6862.12-1.132.el6
provider:glibc.i6862.12-1.132.el6_5.1
provider:glibc.i6862.12-1.132.el6_5.2
dependency:libICE.so.6
provider:libICE.i6861.0.6-1.el6
dependency:libXaw.so.7
provider:libXaw.i6861.0.11-2.el6
dependency:libtinfo.so.5
provider:ncurses-libs.i6865.7-3.20090208.el6
dependency:libutempter.so.0
provider:libutempter.i6861.1.5-4.1.el6
dependency:/bin/sh
provider:bash.i6864.1.2-15.el6_4
dependency:libc.so.6(GLIBC_2.4)
provider:glibc.i6862.12-1.132.el6
provider:glibc.i6862.12-1.132.el6_5.1
provider:glibc.i6862.12-1.132.el6_5.2
dependency:libXmu.so.6
provider:libXmu.i6861.1.1-2.el6
#
Ifthatdoesn’tsolveyourproblem,youhaveonelasttool:
yumupdate—skip-broken
The —skip-broken option allows you to just ignore the package with the broken
dependency and update the other software packages. This may not help the broken
package,butatleastyoucanupdatetheremainingpackagesonthesystem!
In Table 9.6, the commands to try for broken dependencies with urpm and zypper are
listed.Withzypper,thereisonlytheonecommandtoverifyandfixabrokendependency.
With urpm, if the clean option does not work, you can skip updates on the offensive
package. To do this, you must add the name of the offending package to the file
/etc/urpmi/skip.list.
Table9.6BrokenDependencieswithzypperandurpm
FrontEndTool
Command
urpm
urpmi—clean
zypper
zypperverify
yumrepositories
Justlikethe aptitudesystems,yumhasitssoftwarerepositoriessetupatinstallation.For
most purposes, these pre-installed repositories work just fine for your needs. But if and
whenthetimecomesthatyouneedtoinstallsoftwarefromadifferentrepository,hereare
somethingsyouneedtoknow.
Tip
Awisesystemadministratorstickswithapprovedrepositories.Anapproved
repositoryisonethatissanctionedbythedistribution’sofficialsite.Ifyoustart
addingunapprovedrepositories,youlosetheguaranteeofstability.Andyouwillbe
headingintobrokendependenciesterritory
To see what repositories you are currently pulling software from, type the following
command:
yumrepolist
Ifyoudon’tfindarepositoryyouneedsoftwarefrom,youneedtodoalittleconfiguration
fileediting.Theyumrepositorydefinitionfilesarelocatedin/etc/yum.repos.d.Youneed
toaddtheproperURLandgainaccesstoanynecessaryencryptionkeys.
Goodrepositorysitessuchasrpmfusion.org lay out all the steps necessary to use them.
Sometimes,theserepositorysitesofferanrpmfilethatyoucandownloadandinstallusing
the yum localinstallcommand.Theinstallationofthe rpmfiledoesalltherepository
setupworkforyou.Nowthat’sconvenient!
urpmcallsitsrepositoriesmedia.Thecommandsforlookingat urpmmediaand zypper‘s
repositoriesareinTable9.7.Noticewithbothofthesefront-endtoolsthatyoudonotedit
aconfigurationfile.Instead,toaddmediaorarepository,youjusttypethecommand.
Table9.7zypperandurpmRepositories
Action
Front-EndTool
Command
urpm
urpmq—list-media
Displayrepository
urpm
urpmi.addmediapath_name
Addrepository
zypper
zypperrepos
Displayrepository
zypper
zypperaddrepopath_name
Addrepository
Both Debian–based and Red Hat–based systems use Package Management Systems to
ease the process of managing software. Now we are going to step out of the world of
Package Management Systems and look at something a little more difficult: installing
directlyfromsourcecode.
InstallingfromSourceCode
Chapter4discussedtarballpackages—howtocreatethemusingthe tarcommandline
commandandhowtounpackthem.Beforethefancy rpmand dpkgtools,administrators
hadtoknowhowtounpackandinstallsoftwarefromtarballs.
If you work in the open source software environment much, there’s a good chance you
willstillfindsoftwarepackedupasatarball.Thissectionwalksyouthroughtheprocess
ofunpackingandinstallingatarballsoftwarepackage.
Forthisexample,thesoftwarepackagesysstatisused.Thesysstatutilityisaverynice
softwarepackagethatprovidesavarietyofsystemmonitoringtools.
First,youneedtodownloadthesysstattarballtoyourLinuxsystem.Youcanoftenfind
the sysstatpackageavailableondifferentLinuxsites,butit’susuallybesttogostraight
to the source of the program. In this case, it’s the website
http://sebastien.godard.pagesperso-orange.fr/.
IfyouclicktheDownloadlink,yougotothepagethatcontainsthefilesfordownloading.
Thecurrentversionatthetimeofthiswritingis11.1.1,andthedistributionfilenameis
sysstat-11.1.1.tar.gz.
ClickthelinktodownloadthefiletoyourLinuxsystem.Afteryouhavedownloadedthe
file,youcanunpackit.
Tounpackasoftwaretarball,usethestandardtarcommand:
#
#tar-zxvfsysstat-11.1.1.tar.gz
sysstat-11.1.1/
sysstat-11.1.1/cifsiostat.c
sysstat-11.1.1/FAQ
sysstat-11.1.1/ioconf.h
sysstat-11.1.1/rd_stats.h
sysstat-11.1.1/COPYING
sysstat-11.1.1/common.h
sysstat-11.1.1/sysconfig.in
sysstat-11.1.1/mpstat.h
sysstat-11.1.1/rndr_stats.h
[…]
sysstat-11.1.1/activity.c
sysstat-11.1.1/sar.c
sysstat-11.1.1/iostat.c
sysstat-11.1.1/rd_sensors.c
sysstat-11.1.1/prealloc.in
sysstat-11.1.1/sa2.in
#
#
Nowthatthetarballisunpackedandthefileshaveneatlyputthemselvesintoadirectory
calledsysstat-11.1.1,youcandivedownintothatdirectoryandcontinue.
First, use the cd command to get into the new directory and list the contents of the
directory:
$cdsysstat-11.1.1
$ls
activity.ciconfigprealloc.insa.h
buildINSTALLpr_stats.csar.c
CHANGESioconf.cpr_stats.hsa_wrap.c
cifsiostat.cioconf.hrd_sensors.csysconfig.in
cifsiostat.hiostat.crd_sensors.hsysstat-11.1.1.lsm
common.ciostat.hrd_stats.csysstat-11.1.1.spec
common.hjson_stats.crd_stats.hsysstat.in
configurejson_stats.hREADMEsysstat.ioconf
configure.inMakefile.inrndr_stats.csysstat.service.in
contribmanrndr_stats.hsysstat.sysconfig.in
COPYINGmpstat.csa1.inversion.in
count.cmpstat.hsa2.inxml
count.hnfsiostat-sysstat.csa_common.cxml_stats.c
CREDITSnfsiostat-sysstat.hsadc.cxml_stats.h
cronnlssadf.c
FAQpidstat.csadf.h
format.cpidstat.hsadf_misc.c
$
In the listing of the directory, you should typically see a READMEor AAAREADMEfile.Itis
veryimportanttoreadthisfile.Theactualinstructionsyouneedtofinishthesoftware’s
installationareinthisfile.
Followingtheadvicecontainedinthe READMEfile,thenextstepisto configuresysstat
for your system. This checks your Linux system to ensure it has the proper library
dependencies,inadditiontothepropercompilertocompilethesourcecode:
#./configure
Checkprograms:
.
checkingforgcc…gcc
checkingwhethertheCcompilerworks…yes
checkingforCcompilerdefaultoutputfilename…a.out
[…]
checkingforANSICheaderfiles…(cached)yes
checkingfordirent.hthatdefinesDIR…yes
checkingforlibrarycontainingopendir…nonerequired
checkingctype.husability…yes
checkingctype.hpresence…yes
checkingforctype.h…yes
checkingerrno.husability…yes
checkingerrno.hpresence…yes
checkingforerrno.h…yes
[…]
Checklibraryfunctions:
.
checkingforstrchr…yes
checkingforstrcspn…yes
checkingforstrspn…yes
checkingforstrstr…yes
checkingforsensorssupport…yes
checkingforsensors_get_detected_chipsin-lsensors…no
checkingforsensorslib…no
.
Checksystemservices:
.
checkingforspecialCcompileroptionsneededforlargefiles…no
checkingfor_FILE_OFFSET_BITSvalueneededforlargefiles…64
.
Checkconfiguration:
[…]
Nowcreatefiles:
[…]
config.status:creatingMakefile
Sysstatversion:11.1.1
Installationprefix:/usr/local
rcdirectory:/etc/rc.d
Initdirectory:/etc/rc.d/init.d
Systemdunitdir:
Configurationdirectory:/etc/sysconfig
Manpagesdirectory:${datarootdir}/man
Compiler:gcc
Compilerflags:-g-O2
#
If anything does go wrong, the configure step displays an error message explaining
what’s missing. If you don’t have the GNU C compiler installed in your Linux
distribution, you get a single error message, but for all other issues you should see
multiplemessagesindicatingwhat’sinstalledandwhatisn’t.
The next stage is to build the various binary files using the make command. The make
commandcompilesthesourcecodeandthenthelinkertocreatethefinalexecutablefiles
for the package. As with the configure command, the make command produces lots of
outputasitgoesthroughthestepsofcompilingandlinkingallthesourcecodefiles:
#make
–gcc-osadc.o-c-g-O2-Wall-Wstrict-prototypes-pipe-O2
-DSA_DIR="/var/log/sa"-DSADC_PATH="/usr/local/lib/sa/sadc"
-DUSE_NLS-DPACKAGE="sysstat"
-DLOCALEDIR="/usr/local/share/locale"sadc.c
gcc-oact_sadc.o-c-g-O2-Wall-Wstrict-prototypes-pipe-O2
-DSOURCE_SADC-DSA_DIR="/var/log/sa"
-DSADC_PATH="/usr/local/lib/sa/sadc"
-DUSE_NLS-DPACKAGE="sysstat"
-DLOCALEDIR="/usr/local/share/locale"activity.c
[…]
#
When make is finished, you have the actual sysstat software program available in the
directory! However, it’s somewhat inconvenient to have to run it from that directory.
Instead,you’llwanttoinstallitinacommonlocationonyourLinuxsystem.Todothat,
you need to log in as the root user account (or use the sudo command if your Linux
distributionprefers)andthenusetheinstalloptionofthemakecommand:
#makeinstall
mkdir-p/usr/local/share/man/man1
mkdir-p/usr/local/share/man/man5
mkdir-p/usr/local/share/man/man8
rm-f/usr/local/share/man/man8/sa1.8*
install-m644-gmanman/sa1.8/usr/local/share/man/man8
rm-f/usr/local/share/man/man8/sa2.8*
install-m644-gmanman/sa2.8/usr/local/share/man/man8
rm-f/usr/local/share/man/man8/sadc.8*
[…]
install-m644-gmanman/sadc.8/usr/local/share/man/man8
install-m644FAQ/usr/local/share/doc/sysstat-11.1.1
install-m644*.lsm/usr/local/share/doc/sysstat-11.1.1
#
Nowthe sysstat package is installed on the system! Although it’s not quite as easy as
installing a software package via a PMS, installing software using tarballs is not that
difficult.
Summary
This chapter discussed how to work with a Package Management Systems (PMS) to
install,update,orremovesoftwarefromthecommandline.AlthoughmostoftheLinux
distributionsusefancyGUItoolsforsoftwarepackagemanagement,youcanalsoperform
packagemanagementfromthecommandline.
TheDebian-basedLinuxdistributionsusethe dpkgutilitytointerfacewiththePMSfrom
the command line. A front-end to the dpkg utility is aptitude. It provides simple
commandlineoptionsforworkingwithsoftwarepackagesinthedpkgformat.
TheRedHat–basedLinuxdistributionsarebasedontherpmutilitybutusedifferentfrontendtoolsatthecommandline.RedHatandFedorause yumforinstallingandmanaging
softwarepackages.TheopenSUSEdistributionuseszypperformanagingsoftware,while
theMandrivadistributionusesurpm.
The chapter closed with a discussion on how to install software packages that are only
distributed in source code tarballs. The tar command allows you to unpack the source
codefilesfromthetarball,andconfigureandmakeallowyoutobuildthefinalexecutable
programfromthesourcecode.
ThenextchapterlooksatthedifferenteditorsavailableinLinuxdistributions.Asyouget
ready to start working on shell scripts, it will come in handy to know what editors are
availabletouse!
Chapter10
WorkingwithEditors
InThisChapter
1. Workingwiththevimeditor
2. Exploringnano
3. Understandingemacs
4. Gettingcomfortablewithkwrite
5. LookingatKate
6. UsingtheGNOMEeditor
Beforeyoucanstartyourshellscriptingcareer,youneedtoknowhowtouseatleast
onetexteditorinLinux.Themoreyouknowabouthowtousefeaturessuchas
searching,cutting,andpasting,thequickeryoucandevelopyourshellscripts.
Youcanchoosefromseveraleditors.Manyindividualsfindaparticulareditorwhose
featurestheyloveandexclusivelyusethattexteditor.Thischapterdiscussesjusta
fewofthetexteditorsyou’llseeintheLinuxworld.
VisitingthevimEditor
ThevieditorwastheoriginaleditorusedonUnixsystems.Itusedtheconsolegraphics
modetoemulateatext-editingwindow,allowingyoutoseethelinesofyourfile,move
aroundwithinthefile,andinsert,edit,andreplacetext.
Although it was quite possibly the most complicated editor in the world (at least in the
opinionofthosewhohateit),itprovidesmanyfeaturesthathavemadeitastapleforUnix
administratorsfordecades.
WhentheGNUProjectportedthevieditortotheopensourceworld,theychosetomake
someimprovementstoit.Becauseitnolongerresembledtheoriginalvieditorfoundin
theUnixworld,thedevelopersalsorenamedit,toviimproved,orvim.
Thissectionwalksyouthroughthebasicsofusingthevimeditortoedityourtextshell
scriptfiles.
Checkingyourvimpackage
Beforeyoubeginyourexplorationofthevimeditor,it’sagoodideatounderstandwhat
vim package your Linux system has installed. On some distributions, you will have the
full vim package installed and an alias for the vi command, as shown on this CentOS
distribution:
$aliasvi
aliasvi=‘vim’
$
$whichvim
/usr/bin/vim
$
$ls-l/usr/bin/vim
-rwxr-xr-x.1rootroot1967072Apr52012/usr/bin/vim
$
Noticethattheprogramfile’slonglistingdoesnotshowanylinkedfiles(seeChapter3
formoreinformationonlinkedfiles).Ifthevimprogramislinked,itmaybelinkedtoa
lessthanfull-featurededitor.Thus,it’sagoodideatocheckforlinkedfiles.
On other distributions, you will find various flavors of the vim editor. Notice on this
Ubuntu distribution that not only is there no alias for the vi command, but the
/usr/bin/viprogramfilebelongstoaseriesoffilelinks:
$aliasvi
-bash:alias:vi:notfound
$
$whichvi
/usr/bin/vi
$
$ls-l/usr/bin/vi
lrwxrwxrwx1rootroot20Apr2212:39
/usr/bin/vi->/etc/alternatives/vi
$
$ls-l/etc/alternatives/vi
lrwxrwxrwx1rootroot17Apr2212:33
/etc/alternatives/vi->/usr/bin/vim.tiny
$
$ls-l/usr/bin/vim.tiny
-rwxr-xr-x1rootroot884360Jan214:40
/usr/bin/vim.tiny
$
$readlink-f/usr/bin/vi
/usr/bin/vim.tiny
Thus,whenthevicommandisentered,the/usr/bin/vim.tinyprogramisexecuted.The
vim.tinyprogramprovidesonlyafewvimeditorfeatures.Ifyouareseriousaboutusing
thevimeditorandhaveUbuntu,youshouldinstallatleastthebasicvimpackage.
Note
Noticeintheprecedingexamplethat,insteadofhavingtousethels-lcommand
multipletimestofindaseriesoflinkedfiles’finalobject,youcanusethereadlink
-fcommand.Itimmediatelyproducesthelinkedfileseries’finalobject.
Software installations were covered in detail in Chapter 9. Installing the basic vim
packageonthisUbuntudistributionisfairlystraightforward:
$sudoapt-getinstallvim
[…]
Thefollowingextrapackageswillbeinstalled:
vim-runtime
Suggestedpackages:
ctagsvim-docvim-scripts
ThefollowingNEWpackageswillbeinstalled:
vimvim-runtime
[…]
$
$readlink-f/usr/bin/vi
/usr/bin/vim.basic
$
The basic vim editor is now installed on this Ubuntu distribution, and the /usr/bin/vi
program file’s link was automatically changed to point to /usr/bin/vim.basic. Thus,
when the vi command is entered on this Ubuntu system, the basic vim editor is used
insteadoftinyvim.
Exploringvimbasics
Thevimeditorworkswithdatainamemorybuffer.Tostartthevimeditor,justtypethe
vimcommand(orviifthere’sanaliasorlinkedfile)andthenameofthefileyouwantto
edit:
$vimmyprog.c
Ifyoustartvimwithoutafilename,orifthefiledoesn’texist,vimopensanewbufferarea
forediting.Ifyouspecifyanexistingfileonthecommandline,vimreadstheentirefile’s
contentsintoabufferarea,whereitisreadyforediting,asshowninFigure10.1.
Figure10.1Thevimmainwindow
The vim editor detects the terminal type for the session (see Chapter 2) and uses a fullscreenmodetousetheentireconsolewindowfortheeditorarea.
Theinitialvimeditwindowshowsthecontentsofthefile(ifthereareany)alongwitha
message line at the bottom of the window. If the file contents don’t take up the entire
screen,vimplacesatildeonlinesthatarenotpartofthefile(asshowninFigure10.1).
Themessagelineatthebottomindicatesinformationabouttheeditedfile,dependingon
the file’s status, and the default settings in your vim installation. If the file is new, the
message[NewFile]appears.
Thevimeditorhastwomodesofoperation:
Normalmode
Insertmode
Whenyoufirstopenafile(orstartanewfile)forediting,thevimeditorentersnormal
mode.Innormalmode,thevimeditorinterpretskeystrokesascommands(moreonthose
later).
Ininsertmode,viminsertseverykeyyoutypeatthecurrentcursorlocationinthebuffer.
Toenterinsertmode,presstheikey.Togetoutofinsertmodeandgobackintonormal
mode,presstheEscapekeyonthekeyboard.
Innormalmode,youcanmovethecursoraroundthetextareabyusingthearrowkeys(as
long as your terminal type is detected properly by vim). If you happen to be on a flaky
terminalconnectionthatdoesn’thavethearrowkeysdefined,allhopeisnotlost.Thevim
commandsincludecommandsformovingthecursor:
htomoveleftonecharacter
jtomovedownoneline(thenextlineinthetext)
ktomoveuponeline(thepreviouslineinthetext)
ltomoverightonecharacter
Moving around within large text files line by line can get tedious. Fortunately, vim
providesafewcommandstohelpspeedthingsalong:
PageDown(orCtrl+F)tomoveforwardonescreenofdata
PageUp(orCtrl+B)tomovebackwardonescreenofdata
Gtomovetothelastlineinthebuffer
numGtomovetothelinenumbernuminthebuffer
ggtomovetothefirstlineinthebuffer
Thevimeditorhasaspecialfeaturewithinnormalmodecalledcommandlinemode.The
commandlinemodeprovidesaninteractivecommandlinewhereyoucanenteradditional
commands to control the actions in vim. To get to command line mode, press the colon
key in normal mode. The cursor moves to the message line, and a colon (:) appears,
waitingforyoutoenteracommand.
Withinthecommandlinemodeareseveralcommandsforsavingthebuffertothefileand
exitingvim:
qtoquitifnochangeshavebeenmadetothebufferdata
q!toquitanddiscardanychangesmadetothebufferdata
wfilenametosavethefileunderadifferentfilename
wqtosavethebufferdatatothefileandquit
After seeing just a few basic vim commands, you might understand why some people
absolutelyhatethevimeditor.Tobeabletousevimtoitsfullest,youmustknowplenty
of obscure commands. However, after you get a few of the basic vim commands down,
you can quickly edit files directly from the command line, no matter what type of
environmentyou’rein.Plus,afteryougetcomfortabletypingcommands,italmostseems
secondnaturetotypebothdataandeditingcommands,anditbecomesoddhavingtojump
backtousingamouse!
Editingdata
Whileininsertmode,youcaninsertdataintothebuffer;however,sometimesyouneedto
addorremovedataafteryou’vealreadyentereditintothebuffer.Whileinnormalmode,
the vim editor provides several commands for editing the data in the buffer. Table10.1
listssomecommoneditingcommandsforvim.
Table10.1vimEditingCommands
Command
x
dd
dw
d$
J
u
a
A
rchar
Rtext
Description
Deletesthecharacteratthecurrentcursorposition
Deletesthelineatthecurrentcursorposition
Deletesthewordatthecurrentcursorposition
Deletestotheendofthelinefromthecurrentcursorposition
Deletesthelinebreakattheendofthelineatthecurrentcursorposition
(joinslines)
Undoesthepreviouseditcommand
Appendsdataafterthecurrentcursorposition
Appendsdatatotheendofthelineatthecurrentcursorposition
Replacesasinglecharacteratthecurrentcursorpositionwithchar
Overwritesthedataatthecurrentcursorpositionwithtext,untilyoupress
Escape
Someoftheeditingcommandsalsoallowyoutouseanumericmodifiertoindicatehow
many times to perform the command. For example, the command 2x deletes two
characters, starting from the current cursor position, and the command 5dd deletes five
lines,startingatthelinefromthecurrentcursorposition.
Note
BecarefulwhentryingtousethekeyboardBackspaceorDeletekeyswhileinthe
vimeditor’snormalmode.ThevimeditorusuallyrecognizestheDeletekeyasthe
functionalityofthexcommand,deletingthecharacteratthecurrentcursorlocation.
Usually,thevimeditordoesn’trecognizetheBackspacekeyinnormalmode
Copyingandpasting
A standard editor feature is the ability to cut or copy data and paste it elsewhere in the
document.Thevimeditorprovidesawaytodothis.
Cutting and pasting is relatively easy. You’ve already seen the commands in Table10.1
thatcanremovedatafromthebuffer.However,whenvimremovesdata,itactuallykeeps
itstoredinaseparateregister.Youcanretrievethatdatabyusingthepcommand.
Forexample,youcanusethe ddcommandtodeletealineoftext,movethecursortothe
bufferlocationwhereyouwanttoplaceit,andthenusethe pcommand.The pcommand
inserts the text after the line at the current cursor position. You can do this with any
commandthatremovestext.
Copyingtextisalittlebittrickier.Thecopycommandinvimisy(foryank).Youcanuse
thesamesecondcharacterwithyaswiththedcommand(ywtoyankaword,y$toyankto
theendofaline).Afteryouyankthetext,movethecursortothelocationwhereyouwant
toplacethetextandusethepcommand.Theyankedtextnowappearsatthatlocation.
Yankingistrickyinthatyoucan’tseewhathappenedbecauseyou’renotaffectingthetext
thatyouyank.Youneverknowforsurewhatyouyankeduntilyoupasteitsomewhere.
Butthere’sanotherfeatureinvimthathelpsyououtwithyanking.
Thevisualmodehighlightstextasyoumovethecursor.Youusevisualmodetoselecttext
toyankforpasting.Toentervisualmode,movethecursortothelocationwhereyouwant
tostartyanking,andpressv.Noticethatthetextatthecursorpositionisnowhighlighted.
Next,movethecursortocoverthetextyouwanttoyank(youcanevenmovedownlines
toyankmorethanonelineoftext).Asyoumovethecursor,vimhighlightsthetextinthe
yankarea.Afteryou’vecoveredthetextyouwanttocopy,presstheykeytoactivatethe
yankcommand.Nowthatyouhavethetextintheregister,justmovethecursortowhere
youwanttopasteandusethepcommand.
Searchingandsubstituting
You can easily search for data in the buffer using the vim search command. To enter a
searchstring,presstheforwardslash(/)key.Thecursorgoestothemessageline,andvim
displaysaforwardslash.Enterthetextyouwanttofind,andpresstheEnterkey.Thevim
editorrespondswithoneofthreeactions:
Ifthewordappearsafterthecurrentcursorlocation,itjumpstothefirstlocation
wherethetextappears.
Iftheworddoesn’tappearafterthecurrentcursorlocation,itwrapsaroundtheendof
thefiletothefirstlocationinthefilewherethetextappears(andindicatesthiswitha
message).
Itproducesanerrormessagestatingthatthetextwasnotfoundinthefile.
Tocontinuesearchingforthesameword,presstheforwardslashcharacterandthenpress
theEnterkey,oryoucanusethenkey,fornext.
Thesubstitutecommandallowsyoutoquicklyreplace(substitute)onewordforanotherin
the text. To get to the substitute command, you must be in command line mode. The
formatforthesubstitutecommandis:
:s/old/new/
Thevimeditorjumpstothefirstoccurrenceofthetext oldandreplacesitwiththetext
new.Youcanmakeafewmodificationstothesubstitutecommandtosubstitutemorethan
oneoccurrenceofthetext:
:s/old/new/gtoreplacealloccurrencesofoldinaline
:n,ms/old/new/gtoreplacealloccurrencesofoldbetweenlinenumbersnandm
:%s/old/new/gtoreplacealloccurrencesofoldintheentirefile
:%s/old/new/gctoreplacealloccurrencesofoldintheentirefile,butpromptfor
eachoccurrence
As you can see, for a console mode text editor, vim contains quite a few advanced
features.BecauseeveryLinuxdistributionincludesit,it’sagoodideatoatleastknowthe
basicsofthevimeditorsoyoucanalwayseditscripts,nomatterwhereyouareorwhat
youhaveavailable.
NavigatingthenanoEditor
Although vim is a very complicated editor with many powerful features, nano is a very
simpleeditor.Forindividualswhoneedasimpleconsolemodetexteditorthatiseasyto
navigate,nanoisthetooltouse.It’salsoagreattexteditorforkidswhoarestartingon
theirLinuxcommandlineadventure.
ThenanotexteditorisacloneoftheUnixsystems’Picoeditor.AlthoughPicoalsoisa
light and simple text editor, it is not licensed under the GPL. Not only is the nano text
editorlicensedundertheGPL,itisalsopartoftheGNUproject.
ThenanotexteditorisinstalledonmostLinuxdistributionsbydefault.Everythingabout
thenanotexteditorissimple.Toopenafileatthecommandlinewithnano:
$nanomyprog.c
Ifyoustartnanowithoutafilename,orifthefiledoesn’texist,nanosimplyopensanew
bufferareaforediting.Ifyouspecifyanexistingfileonthecommandline,nanoreadsthe
entire contents of the file into a buffer area, where it is ready for editing, as shown in
Figure10.2.
Figure10.2Thenanoeditorwindow
Noticeatthebottomofthenanoeditorwindowvariouscommandswithabriefdescription
areshown.Thesecommandsarethenanocontrolcommands.Thecaret(∧)symbolshown
representstheCtrlkey.Therefore,∧XstandsforthekeyboardsequenceCtrl+X.
Tip
Thoughthenanocontrolcommandslistcapitallettersinthekeyboardsequences,you
canuseeitherlowercaseoruppercasecharactersforcontrolcommands.
Havingallthebasiccommandslistedrightinfrontofyouisgreat.Noneedtomemorize
what control command does what. Table 10.2 presents the various nano control
commands.
Table10.2nanoControlCommands
Command
Description
CTRL+C
Displaysthecursor’spositionwithinthetexteditingbuffer
CTRL+G
Displaysnano’smainhelpwindow
CTRL+J
Justifiesthecurrenttextparagraph
CTRL+K
Cutsthetextlineandstoresitincutbuffer
CTRL+O
Writesoutthecurrenttexteditingbuffertoafile
CTRL+R
Readsafileintothecurrenttexteditingbuffer
CTRL+T
Startstheavailablespellchecker
CTRL+U
Pastestextstoredincutbufferandplacesincurrentline
CTRL+V
Scrollstexteditingbuffertonextpage
CTRL+W
Searchesforwordorphraseswithintexteditingbuffer
CTRL+X Closesthecurrenttexteditingbuffer,exitsnano,andreturnstotheshell
CTRL+Y
Scrollstexteditingbuffertopreviouspage
ThecontrolcommandslistedinTable10.2arereallyallyouneed.However,ifyoudesire
more powerful control features than those listed, nano has them. To see more control
commands,typeCtrl+Ginthenanotexteditortodisplayitsmainhelpwindowcontaining
additionalcontrolcommands.
Note
IfyoutrytousethenanospellcheckerviatheCtrl+Tcommandandgettheerror
messageSpellcheckingfailed:Errorinvoking‘Spell’,therearesome
potentialsolutions.Installthespellcheckersoftwarepackage,aspell,onyourLinux
distributionusingChapter9asaguide.
Ifinstallingtheaspellsoftwarepackagedoesnotsolvetheproblem,assuperuser
editthe/etc/nanorcfile,usingyourfavoritetexteditor.Findtheline,#set
speller“aspell-x-c”anddeletethehashmark(#)fromtheline.Saveandexit
thefile.
Additional powerful features are available at the command line. You can use command
lineoptionstocontrolnanoeditorfeatures,suchascreatingabackupfilebeforeediting.
Typemannanotoseetheseadditionalcommandlineoptionsforstartingnano.
Thevimandnanotexteditorsofferachoicebetweenpowerfulandsimpleconsolemode
texteditors.However,neitherofferstheabilitytousegraphicalfeaturesforediting.Some
texteditorscanoperateinbothworlds,asexploredinthenextsection.
ExploringtheemacsEditor
The emacs editor is an extremely popular editor that appeared before even Unix was
around.DeveloperslikeditsomuchthattheyportedittotheUnixenvironment,andnow
it’sbeenportedtotheLinuxenvironment.Theemacseditorstartedoutlifeasaconsole
editor,muchlikevi,buthasmigratedtothegraphicalworld.
Theemacseditorstillprovidestheoriginalconsolemodeeditor,andnowitalsohasthe
ability to use a graphical window to allow editing text in a graphical environment.
Typically, when you start the emacs editor from a command line, the editor determines
whetheryouhaveanavailablegraphicalsessionandstartsingraphicalmode.Ifyoudon’t,
itstartsinconsolemode.
This section describes both the console mode and graphical mode emacs editors so that
you’llknowhowtouseeitheroneifyouwant(orneed)to.
Checkingyouremacspackage
Manydistributionsdonotcomewiththeemacseditorinstalledbydefault.Youcancheck
yourRedHat-baseddistribution,byusingthewhichand/oryumlistcommandasshown
onthisCentOSdistribution:
$whichemacs
/usr/bin/which:noemacsin(/usr/lib64/qt-3.3
/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:
/usr/sbin:/sbin:/home/Christine/bin)
$
$yumlistemacs
[…]
AvailablePackages
emacs.x86_641:23.1-25.el6base
TheemacseditorpackageisnotcurrentlyinstalledonthisCentOSdistribution.However,
it is available to be installed. (For a more thorough discussion on displaying installed
software,seeChapter9).
ForaDebian-baseddistribution,checkfortheemacseditorpackagebyusingthe which
and/orapt-cacheshowcommandasshownonthisUbuntudistribution:
$whichemacs
$
$sudoapt-cacheshowemacs
Package:emacs
Priority:optional
Section:editors
Installed-Size:25
[…]
Description-en:GNUEmacseditor(metapackage)
GNUEmacsistheextensibleself-documentingtexteditor.
Thisisametapackagethatwillalwaysdependonthelatest
recommendedEmacsrelease.
Description-md5:21fb7da111336097a2378959f6d6e6a8
Bugs:https://bugs.launchpad.net/ubuntu/+filebug
Origin:Ubuntu
Supported:5y
$
The whichcommandoperatesalittledifferentlyhere.Whenitdoesnotfindtheinstalled
command,itsimplyreturnsthebashshellprompt.Theemacseditorpackageisoptional
for this Ubuntu distribution, but is available to be installed. The following shows the
emacseditorbeinginstalledonUbuntu:
$sudoapt-getinstallemacs
Readingpackagelists…Done
Buildingdependencytree
Readingstateinformation…Done
Thefollowingextrapackageswillbeinstalled:
[…]
Installemacsen-commonforemacs24
emacsen-common:Handlinginstallofemacsenflavoremacs24
Wrote/etc/emacs24/site-start.d/00debian-vars.elc
Wrote/usr/share/emacs24/site-lisp/debian-startup.elc
Settingupemacs(45.0ubuntu1)…
Processingtriggersforlibc-bin(2.19-0ubuntu6)…
$
$whichemacs
/usr/bin/emacs
$
Nowwhenthe which command is used, it points to the emacs program file. The emacs
editorisreadytobeusedonthisUbuntudistribution.
FortheCentOSdistribution,installtheemacseditorusingtheyuminstallcommand:
$sudoyuminstallemacs
[sudo]passwordforChristine:
[…]
SettingupInstallProcess
ResolvingDependencies
[…]
Installed:
emacs.x86_641:23.1-25.el6
DependencyInstalled:
emacs-common.x86_641:23.1-25.el6
libotf.x86_640:0.9.9-3.1.el6
m17n-db-datafiles.noarch0:1.5.5-1.1.el6
Complete!
$
$whichemacs
/usr/bin/emacs
$
$yumlistemacs
[…]
InstalledPackages
emacs.x86_641:23.1-25.el6@base
$
WiththeemacseditorsuccessfullyinstalledonyourLinuxdistribution,youcanbeginto
exploreitsdifferentfeatures,staringwithusingitontheconsole.
Usingemacsontheconsole
Theconsolemodeversionofemacsisanothereditorthatuseslotsofkeycommandsto
performeditingfunctions.TheemacseditoruseskeycombinationsinvolvingtheControl
key(theCtrlkeyonthekeyboard)andtheMetakey.Inmostterminalemulatorpackages,
theMetakeyismappedtotheAltkey.TheofficialemacsdocumentsabbreviatetheCtrl
key as C- and the Meta key as M-. Thus, if you enter a Ctrl+x key combination, the
documentshowsC-x.Thischapterdoesthesamesoasnottoconfuseyou.
Exploringthebasicsofemacs
Toeditafileusingemacs,fromthecommandline,enter:
$emacsmyprog.c
Theemacsconsolemodewindowappearswithashortintroductionandhelpscreen.Don’t
be alarmed; as soon as you press a key, emacs loads the file into the active buffer and
displaysthetext,asshowninFigure10.3.
Figure10.3Editingafileusingtheemacseditorinconsolemode
You’ll notice that the top of the console mode window shows a typical menu bar.
Unfortunately,youcan’tusethemenubarinconsolemode,onlyingraphicalmode.
Note
Somecommandsinthissectionworkdifferentlythandescribed,ifyourunemacsin
agraphicaldesktopenvironment.Touseemac’sconsolemodeinagraphicaldesktop
environment,usetheemacs-nwcommand.Ifyouwanttouseemacs’graphical
features,seethesection“UsingemacsinaGUI.”
Unlike the vim editor, where you have to move into and out of insert mode to switch
between entering commands and inserting text, the emacs editor has only one mode. If
youtypeaprintablecharacter,emacsinsertsitatthecurrentcursorposition.Ifyoutypea
command,emacsexecutesthecommand.
Tomovethecursoraroundthebufferarea,youcanusethearrowkeysandthePageUp
and PageDown keys, assuming that emacs detected your terminal emulator correctly. If
not,thesecommandsmovethecursoraround:
C-pmovesuponeline(thepreviouslineinthetext).
C-bmovesleft(back)onecharacter.
C-fmovesright(forward)onecharacter.
C-nmovesdownoneline(thenextlineinthetext).
Thesecommandsmakelongerjumpswiththecursorwithinthetext:
M-fmovesright(forward)tothenextword.
M-bmovesleft(backward)tothepreviousword.
C-amovestothebeginningofthecurrentline.
C-emovestotheendofthecurrentline.
M-amovestothebeginningofthecurrentsentence.
M-emovestotheendofthecurrentsentence.
M-vmovesbackonescreenofdata.
C-vmovesforwardonescreenofdata.
M-<movesthefirstlineofthetext.
M->movestothelastlineofthetext.
You should know these commands for saving the editor buffer back into the file and
exitingemacs:
C-xC-ssavesthecurrentbuffercontentstothefile.
C-zexitsemacsbutkeepsitrunninginyoursessionsoyoucancomebacktoit.
C-xC-cexitsemacsandstopstheprogram.
You’llnoticethattwoofthesefeaturesrequiretwokeycommands.The C-xcommandis
called the extendcommand. This provides yet another whole set of commands to work
with.
Editingdata
Theemacseditorisprettyrobustaboutinsertinganddeletingtextinthebuffer.Toinsert
text,justmovethecursortothelocationwhereyouwanttoinsertthetextandstarttyping.
To delete text, emacs uses the Backspace key to delete the character before the current
cursorpositionandtheDeletekeytodeletethecharacteratthecurrentcursorlocation.
Theemacseditoralsohascommandsforkillingtext.Thedifferencebetweendeletingtext
andkillingtextisthatwhenyoukilltext,emacsplacesitinatemporaryareawhereyou
canretrieveit(seethenextsection,“Copyingandpasting”).Deletedtextisgoneforever.
Thesecommandsareforkillingtextinthebuffer:
M-Backspacekillsthewordbeforethecurrentcursorposition.
M-dkillsthewordafterthecurrentcursorposition.
C-kkillsfromthecurrentcursorpositiontotheendoftheline.
M-kkillsfromthecurrentcursorpositiontotheendofthesentence.
Theemacseditoralsoincludesafancywayofmass-killingtext.Justmovethecursorto
thestartoftheareayouwanttokill,andpresseitherthe C-@or C-Spacebarkeys.Then
movethecursortotheendoftheareayouwanttokill,andpressthe C-wcommandkeys.
Allthetextbetweenthetwolocationsiskilled.
If you happen to make a mistake when killing text, the C-/ command undoes the kill
commandandreturnsthedatatothestateitwasinbeforeyoukilledit.
Copyingandpasting
You’veseenhowtocutdatafromtheemacsbufferarea;nowit’stimetoseehowtopaste
itsomewhereelse.Unfortunately,ifyouusethevimeditor,thisprocessmayconfuseyou
whenyouusetheemacseditor.
Inanunfortunatecoincidence,pastingdatainemacsiscalledyanking.Inthevimeditor,
copyingiscalledyanking,whichiswhatmakesthisadifficultthingtorememberifyou
happentousebotheditors.
Afteryoukilldatausingoneofthekillcommands,movethecursortothelocationwhere
you want to paste the data, and use the C-y command. This yanks the text out of the
temporaryareaandpastesitatthecurrentcursorposition.The C-ycommandyanksthe
text from the last kill command. If you’ve performed multiple kill commands, you can
cyclethroughthemusingtheM-ycommand.
Tocopytext,justyankitbackintothesamelocationyoukilleditfromandthenmoveto
thenewlocationandusethe C-ycommandagain.Youcanyanktextbackasmanytimes
asyoudesire.
SearchingandReplacing
SearchingfortextintheemacseditorisdonebyusingtheC-sandC-rcommands.TheCscommandperformsaforwardsearchinthebufferareafromthecurrentcursorposition
to the end of the buffer, whereas the C-r command performs a backward search in the
bufferareafromthecurrentcursorpositiontothestartofthebuffer.
When you enter either the C-s or C-r command, a prompt appears in the bottom line,
queryingyouforthetexttosearch.Youcanperformtwotypesofsearchesinemacs.
Inanincrementalsearch,theemacseditorperformsthetextsearchinreal-timemodeas
youtypetheword.Whenyoutypethefirstletter,ithighlightsalltheoccurrencesofthat
letterinthebuffer.Whenyoutypethesecondletter,ithighlightsalltheoccurrencesofthe
two-lettercombinationinthetextandsoonuntilyoucompletethetextyou’researching
for.
In a non-incremental search, press the Enter key after the C-s or C-r commands. This
locksthesearchqueryintothebottomlineareaandallowsyoutotypethesearchtextin
fullbeforesearching.
Toreplaceanexistingtextstringwithanewtextstring,youmustusethe M-xcommand.
Thiscommandrequiresatextcommand,alongwithparameters.
The text command is replace-string. After typing the command, press the Enter key,
andemacsqueriesyoufortheexistingtextstring.Afterenteringthat,presstheEnterkey
againandemacsqueriesyouforthenewreplacementtextstring.
Usingbuffersinemacs
The emacs editor allows you to edit multiple files at the same time by having multiple
bufferareas.Youcanloadfilesintoabufferandswitchbetweenbufferswhileediting.
Toloadanewfileintoabufferwhileyou’reinemacs,usethe C-x C-fkeycombination.
This is the emacs Find a File mode. It takes you to the bottom line in the window and
allowsyoutoenterthenameofthefileyouwanttostarttoedit.Ifyoudon’tknowthe
nameorlocationofthefile,justpresstheEnterkey.Thisbringsupafilebrowserinthe
editwindow,asshowninFigure10.4.
Figure10.4TheemacsFindaFilemodebrowser
Fromhere,youcanbrowsetothefileyouwanttoedit.Totraverseupadirectorylevel,go
to the double dot entry and press the Enter key. To traverse down a directory, go to the
directoryentryandpresstheEnterkey.Whenyou’vefoundthefileyouwanttoedit,press
theEnterkeyandemacsloadsitintoanewbufferarea.
You can list the active buffer areas by pressing the C-x C-b extended command
combination.Theemacseditorsplitstheeditorwindowanddisplaysalistofbuffersinthe
bottomwindow.emacsprovidestwobuffersinadditiontoyourmaineditingbuffer:
Ascratchareacalled*scratch*
Amessageareacalled*Messages*
ThescratchareaallowsyoutoenterLISPprogrammingcommandsaswellasenternotes
toyourself.Themessageareashowsmessagesgeneratedbyemacswhileoperating.Ifany
errorsoccurwhileusingemacs,theyappearinthemessagearea.
Youcanswitchtoadifferentbufferareainthewindowintwoways:
UseC-xotoswitchtothebufferlistingwindow.Usethearrowkeystomovetothe
bufferareayouwantandpresstheEnterkey.
UseC-xbtotypeinthenameofthebufferareayouwanttoswitchto.
Whenyouselecttheoptiontoswitchtothebufferlistingwindow,emacsopensthebuffer
areainthenewwindowarea.Theemacseditorallowsyoutohavemultiplewindowsopen
inasinglesession.Thefollowingsectiondiscusseshowtomanagemultiplewindowsin
emacs.
Usingwindowsinconsolemodeemacs
The console mode emacs editor was developed many years before the idea of graphical
windowsappeared.However,itwasadvancedforitstime,inthatitcouldsupportmultiple
editingwindowswithinthemainemacswindow.
You can split the emacs editing window into multiple windows by using one of two
commands:
C-x2splitsthewindowhorizontallyintotwowindows.
C-x3splitsthewindowverticallyintotwowindows.
To move from one window to another, use the C-x o command. Notice that when you
create a new window, emacs uses the buffer area from the original window in the new
window.Afteryoumoveintothenewwindow,youcanusetheC-xC-fcommandtoload
a new file or use one of the commands to switch to a different buffer area in the new
window.
Tocloseawindow,movetoitandusethe C-x0(that’sazero)command.Ifyouwantto
close all the windows except the one you’re in, use the C-x 1 (that’s a numerical one)
command.
UsingemacsinaGUI
If you use emacs from a GUI environment (such as the Unity or GNOME desktops), it
startsingraphicalmode,asshowninFigure10.5.
Figure10.5Theemacsgraphicalwindow
If you’ve already used emacs in console mode, you should be fairly familiar with the
graphicalmode.Allthekeycommandsareavailableasmenubaritems.Theemacsmenu
barcontainsthefollowingitems:
Fileallowsyoutoopenfilesinthewindow,createnewwindows,closewindows,
savebuffers,andprintbuffers.
Editallowsyoutocutandcopyselectedtexttotheclipboard,pasteclipboarddatato
thecurrentcursorposition,searchfortext,andreplacetext.
Optionsprovidessettingsformanymoreemacsfeatures,suchashighlighting,word
wrap,cursortype,andsettingfonts.
Buffersliststhecurrentbuffersavailableandallowsyoutoeasilyswitchbetween
bufferareas.
Toolsprovidesaccesstotheadvancedfeaturesinemacs,suchasthecommandline
interfaceaccess,spellchecking,comparingtextbetweenfiles(calleddiff),sending
ane-mailmessage,calendar,andthecalculator.
Helpprovidestheemacsmanualonlineforaccesstohelponspecificemacs
functions.
Inadditiontothenormalgraphicalemacsmenubaritems,thereisoftenaseparateitem
specifictothefiletypeintheeditorbuffer.Figure10.5showsopeningaCprogram,so
emacsprovidedaCmenuitem,allowingadvancedsettingsforhighlightingCsyntax,and
compiling,running,anddebuggingthecodefromacommandprompt.
The graphical emacs window is an example of an older console application making the
migration to the graphical world. Now that many Linux distributions provide graphical
desktops (even on servers that don’t need them), graphical editors are becoming more
commonplace. Popular Linux desktop environments (such as KDE and GNOME) have
alsoprovidedgraphicaltexteditorsspecificallyfortheirenvironments,whicharecovered
intherestofthischapter.
ExploringtheKDEFamilyofEditors
Ifyou’reusingaLinuxdistributionthatusestheKDEdesktop(seeChapter1),youhavea
coupleofoptionswhenitcomestotexteditors.TheKDEprojectofficiallysupportstwo
populartexteditors:
KWrite:Asingle-screentext-editingpackage
Kate:Afull-featured,multi-windowtext-editingpackage
Bothoftheseeditorsaregraphicaltexteditorsthatcontainmanyadvancedfeatures.The
Kate editor provides more advanced features, plus extra niceties not often found in
standard text editors. This section describes each of the editors and shows some of the
featuresyoucanusetohelpwithyourshellscriptediting.
LookingattheKWriteeditor
ThebasiceditorfortheKDEenvironmentisKWrite.Itprovidessimpleword-processing–
styletextediting,alongwithsupportforcodesyntaxhighlightingandediting.Thedefault
KWriteeditingwindowisshowninFigure10.6.
Figure10.6ThedefaultKWritewindoweditingashellscriptprogram
You can’t tell from Figure 10.6, but the KWrite editor recognizes several types of
programming languages and uses color coding to distinguish constants, functions, and
comments.Also,noticethatthe forloophasaniconthatlinkstheopeningandclosing
braces.Thisiscalledafoldingmarker.Byclickingtheicon,youcancollapsethefunction
intoasingleline.Thisisagreatfeaturewhenworkingthroughlargeapplications.
TheKWriteeditingwindowprovidesfullcutandpastecapabilities,usingthemouseand
thearrowkeys.Asinawordprocessor,youcanhighlightandcut(orcopy)textanywhere
inthebufferareaandpasteitatanyotherplace.
ToeditafileusingKWrite,youcaneitherselectKWritefromtheKDEmenusystemon
yourdesktop(someLinuxdistributionsevencreateaPaneliconforit)orstartitfromthe
commandlineprompt:
$kwritefactorial.sh
The kwrite command has several command line parameters you can use to customize
howitstarts:
—stdincausesKWritetoreaddatafromthestandardinputdeviceinsteadofafile.
—encodingspecifiesacharacterencodingtypetouseforthefile.
—linespecifiesalinenumberinthefiletostartatintheeditorwindow.
—columnspecifiesacolumnnumberinthefiletostartatintheeditorwindow.
TheKWriteeditorprovidesbothamenubarandatoolbaratthetopoftheeditwindow,
allowingyoutoselectfeaturesandchangeconfigurationsettingsoftheKWriteeditor.
Themenubarcontainstheseitems:
Fileloads,saves,prints,andexportstextfromfiles.
Editmanipulatestextinthebufferarea.
Viewmanageshowthetextappearsintheeditorwindow.
Bookmarkshandlepointerstoreturntospecificlocationsinthetext;thisoptionmay
needtobeenabledintheconfigurations.
Toolscontainsspecializedfeaturestomanipulatethetext.
Settingsconfiguresthewaytheeditorhandlestext.
Helpgivesyouinformationabouttheeditorandcommands.
The Edit menu bar item provides commands for all your text-editing needs. Instead of
havingtoremembercryptickeycommands(whichbytheway,KWritealsosupports),you
canjustselectitemsintheEditmenubar,asshowninTable10.3.
Table10.3TheKWriteEditMenuItems
Item
Undo
Redo
Cut
Copy
Description
Reversesthelastactionoroperation
Reversesthelastundoaction
Deletestheselectedtextandplacesitintheclipboard
Copiestheselectedtexttotheclipboard
Insertsthecurrentcontentsoftheclipboardatthecurrentcursor
position
SelectAll
Selectsalltextintheeditor
Deselect
Deselectsanytextthatiscurrentlyselected
Overwrite
Togglesinsertmodetooverwritemode,replacingtextwithnewtyped
Mode
textinsteadofjustinsertingthenewtext
ProducestheFindTextdialogbox,whichallowsyoutocustomizea
Find
textsearch
FindNext
Repeatsthelastfindoperationforwardinthebufferarea
FindPrevious
Repeatsthelastfindoperationbackwardsinthebufferarea
ProducestheReplaceWithdialogbox,whichallowsyoutocustomizea
Replace
textsearchandreplace
FindSelected
Findsthenextoccurrenceoftheselectedtext
FindSelected
Findsthepreviousoccurrenceoftheselectedtext
Backwards
ProducestheGotodialogbox,whichallowsyoutoenteralinenumber.
GotoLine
Thecursormovestothespecifiedline
Paste
TheFindfeaturehastwomodes.Normalmodeperformssimpletextsearchesandpower
searches. Replace mode lets you do advanced searching and replacing if necessary. You
toggle between the two modes using the green arrow in the Find section, as shown in
Figure10.7.
Figure10.7TheKWriteFindsection
The Find power mode allows you to search not only with words, but with a regular
expression (discussed in Chapter 20) for the search. You can use some other options to
customizethesearchaswell,indicating,forexample,whetherornottoperformacasesensitivesearchortolookonlyforwholewordsinsteadoffindingthetextwithinwords.
TheToolsmenubaritemprovidesseveralhandyfeaturesforworkingwiththetextinthe
bufferarea.Table10.4describesthetoolsavailableinKWrite.
Table10.4TheKWriteTools
Tool
ReadOnly
Mode
Encoding
Spelling
Spelling
(fromcursor)
Spellcheck
Selection
Indent
Unindent
Clean
Indentation
Align
Uppercase
Lowercase
Capitalize
JoinLines
WordWrap
Document
Description
Locksthetextsonochangescanbemadewhileintheeditor
Setsthecharactersetencodingusedbythetext
Startsthespell-checkprogramatthestartofthetext
Startsthespell-checkprogramfromthecurrentcursorposition
Startsthespell-checkprogramonlyontheselectedsectionoftext
Increasestheparagraphindentationbyone
Decreasestheparagraphindentationbyone
Returnsallparagraphindentationtotheoriginalsettings
Forcesthecurrentlineortheselectedlinestoreturntothedefault
indentationsettings
Setstheselectedtext,orthecharacteratthecurrentcursorposition,to
uppercase
Setstheselectedtext,orthecharacteratthecurrentcursorposition,to
lowercase
Capitalizesthefirstletteroftheselectedtextorthewordatthecurrent
cursorposition
Combinestheselectedlines,orthelineatthecurrentcursorpositionand
thenextline,intooneline
Enableswordwrappinginthetext.Ifalineextendspasttheeditor
windowedge,thelinecontinuesonthenextline.
Therearelotsoftoolsforasimpletexteditor!
TheSettingsmenuincludestheConfigureEditordialogbox,showninFigure10.8.
Figure10.8TheKWriteConfigureEditordialogbox
The Configuration dialog box uses icons on the left side for you to select the feature in
KWritetoconfigure.Whenyouselectanicon,therightsideofthedialogboxshowsthe
configurationsettingsforthefeature.
The Appearance feature allows you to set several features that control how the text
appears in the text editor window. You can enable word wrap, line numbers (great for
programmers), and the folder markers from here. With the Fonts & Colors feature, you
cancustomizethecompletecolorschemefortheeditor,determiningwhatcolorstomake
eachcategoryoftextintheprogramcode.
LookingattheKateeditor
TheKateeditoristheflagshipeditorfortheKDEProject.Itusesthesametexteditoras
theKWriteapplication(somostofthosefeaturesarethesame),butitincorporateslotsof
otherfeaturesintoasinglepackage.
Tip
IfyoufindthattheKateeditorhasnotbeeninstalledwithyourKDEdesktop
environment,youcaneasilyinstallit(seeChapter9).Thepackagenamethat
containsKateiskdesdk.
WhenyoustarttheKateeditorfromtheKDEmenusystem,thefirstthingyounoticeis
thattheeditordoesn’tstart!Instead,yougetadialogbox,asshowninFigure10.9.
Figure10.9TheKatesessiondialogbox
TheKateeditorhandlesfilesinsessions.Youcanhavemultiplefilesopeninasession,
andyoucanhavemultiplesessionssaved.WhenyoustartKate,itprovidesyouwiththe
choiceofwhichsessiontoreturnto.WhenyoucloseyourKatesession,itremembersthe
documentsyouhadopenanddisplaysthemthenexttimeyoustartKate.Thisallowsyou
to easily manage files from multiple projects by using separate workspaces for each
project.
Afterselectingasession,youseethemainKateeditorwindow,showninFigure10.10.
Figure10.10ThemainKateeditingwindow
The left side frame shows the documents currently open in the session. You can switch
between documents just by clicking the document name. To edit a new file, click the
FilesystemBrowsertabontheleftside.Theleftframeisnowafullgraphicalfilesystem
browser,allowingyoutographicallybrowsetolocateyourfiles.
AgreatfeatureoftheKateeditoristhebuilt-interminalwindow,showninFigure10.11.
Figure10.11TheKatebuilt-interminalwindow
The terminal tab at the bottom of the text editor window starts the built-in terminal
emulator in Kate (using the KDE Konsole terminal emulator). This feature horizontally
splitsthecurrenteditingwindow,creatinganewwindowwithKonsolerunninginit.You
can now enter command line commands, start programs, or check on system settings
without having to leave the editor! To close the terminal window, just type exit at the
commandprompt.
As you can tell from the terminal feature, Kate also supports multiple windows. The
Windowmenubaritem(View)providesoptionstoperformthesetasks:
CreateanewKatewindowusingthecurrentsession
Splitthecurrentwindowverticallytocreateanewwindow
Splitthecurrentwindowhorizontallytocreateanewwindow
Closethecurrentwindow
TosettheconfigurationsettingsinKate,selecttheConfigureKateitemundertheSettings
menubaritem.TheConfigurationdialogbox,showninFigure10.12,appears.
Figure10.12TheKateconfigurationsettingsdialogbox
NoticethattheEditorsettingsareaisexactlythesameasforKWrite.Thisisbecausethe
twoeditorssharethesametexteditorengine.TheApplicationsettingsareaallowsyouto
configuresettingsfortheKateitems,suchascontrollingsessions(showninFigure10.12),
the documents list, and the filesystem browser. Kate also supports external plug-in
applications,whichcanbeactivatedhere.
ExploringtheGNOMEEditor
Ifyou’reworkingonaLinuxsystemusingtheGNOMEorUnitydesktopenvironment,
there’sagraphicaltexteditorthatyoucanuseaswell.Thegedittexteditorisabasictext
editor,withafewadvancedfeaturesthrowninjustforfun.Thissectionwalksyouthrough
thefeaturesofgeditanddemonstrateshowtouseitforyourshellscriptprogramming.
Startinggedit
MostGNOMEdesktopenvironmentsincludegeditintheAccessoriesPanelmenuitem.
FortheUnitydesktopenvironment,gotoDash⇨Searchandtypegedit.Ifyoucan’tfind
gedit via the menu system, you can start it from the command line prompt in a GUI
terminalemulator:
$geditfactorial.shmyprog.c
When you start gedit with multiple files, it loads all the files into separate buffers and
displayseachoneasatabbedwindowwithinthemaineditorwindow,asshowninFigure
10.13.
Figure10.13Thegeditmaineditorwindow
The left frame in the gedit main editor window shows the documents you’re currently
editing. If your gedit doesn’t show the left frame when started, you can press the F9
functionkeyorenableSidePanefromtheViewmenu.
Note
Differentdesktopsmayhavegeditoptionsthatareavailableinslightlydifferent
menulocationsthanshowninthesefigures.Additionaloptionsmayalsobe
available.Consultyourdistribution’sgeditHelpmenuformoreassistance.
Therightsideshowsthetabbedwindowsthatcontainthebuffertext.Ifyouhoveryour
mousepointerovereachtab,adialogboxappears,showingthefullpathnameofthefile,
theMIMEtype,andthecharactersetencodingituses.
Understandingbasicgeditfeatures
Inadditiontotheeditorwindows,geditusesbothamenubarandtoolbarthatallowyouto
setfeaturesandconfiguresettings.Thetoolbarprovidesquickaccesstomenubaritems.
Thesemenubaritemsareavailable:
Filehandlesnewfiles,savesexistingfiles,andprintsfiles.
Editmanipulatestextintheactivebufferareaandsetstheeditorpreferences.
Viewsetstheeditorfeaturestodisplayinthewindowandsetsthetexthighlighting
mode.
Searchfindsandreplacestextintheactiveeditorbufferarea.
Toolsaccessesplug-intoolsinstalledingedit.
Documentsmanagesfilesopeninthebufferareas.
Helpprovidesaccesstothefullgeditmanual.
Thereshouldn’tbeanythingtoosurprisinghere.TheEditmenucontainsthestandardcut,
copy,andpastefunctions,alongwithaneatfeaturethatallowsyoutoeasilyenterthedate
andtimeinthetextinseveraldifferentformats.TheSearchmenuprovidesastandardfind
function,whichproducesadialogboxwhereyoucanenterthetexttofind,alongwiththe
capabilitytoselecthowthefindfeatureshouldwork(matchingcase,matchingthewhole
word, and the search direction). It also provides an incremental search feature, which
worksinreal-timemode,findingtextasyoutypethecharactersoftheword.
Settingpreferences
TheEditmenucontainsaPreferencesitem,whichproducesthegeditPreferencesdialog
box,showninFigure10.14.
Figure10.14TheGNOMEdesktopgeditPreferencesdialogbox
Thisiswhereyoucancustomizetheoperationofthegediteditor.ThePreferencesdialog
boxcontainsfivetabbedareasforsettingthefeaturesandbehavioroftheeditor.
SettingViewpreferences
TheViewtabprovidesoptionsforhowgeditdisplaysthetextintheeditorwindow:
TextWrapping:Determineshowtohandlelonglinesoftextintheeditor.The
Enablingtextwrappingoptionwrapslonglinestothenextlineoftheeditor.TheDo
NotSplitWordsOverTwoLinesoptionpreventstheauto-insertingofhyphensinto
longwords,topreventthembeingsplitbetweentwolines.
LineNumbers:Displayslinenumbersintheleftmarginintheeditorwindow.
CurrentLine:Highlightsthelinewherethecursoriscurrentlypositioned,enabling
youtoeasilyfindthecursorposition.
RightMargin:Enablestherightmarginandallowsyoutosethowmanycolumns
shouldbeintheeditorwindow.Thedefaultvalueis80columns.
BracketMatching:Whenenabled,highlightsbracketpairsinprogrammingcode,
allowingyoutoeasilymatchbracketsinif-thenstatements,forandwhileloops,
andothercodingelementsthatusebrackets.
The line-numbering and bracket-matching features provide an environment for
programmerstotroubleshootcodethat’snotoftenfoundintexteditors.
SettingEditorpreferences
The Editor tab provides options for how the gedit editor handles tabs and indentation,
alongwithhowfilesaresaved:
TabStops:SetsthenumberofspacesskippedwhenyoupresstheTabkey.The
defaultvalueiseight.Thisfeaturealsoincludesacheckboxthat,whenselected,
insertsspacesinsteadofatabskip.
AutomaticIndentation:Whenenabled,causesgedittoautomaticallyindentlinesin
thetextforparagraphsandcodeelements(suchasif-thenstatementsandloops).
FileSaving:Providestwofeaturesforsavingfiles:whetherornottocreateabackup
copyofthefilewhenopenedintheeditwindow,andwhetherornottoautomatically
savethefileatapreselectedinterval.
The auto-save feature is a great way to ensure that your changes are saved on a regular
basistopreventcatastrophesfromcrashesorpoweroutages.
SettingFont&Colorpreferences
TheFont&Colorstaballowsyoutoconfigure(notsurprisingly)twoitems:
Font:Allowsyoutoselectthedefaultfont,ortoselectacustomizedfontandfont
sizefromadialogbox.
ColorScheme:Allowsyoutoselectthedefaultcolorschemeusedfortext,
background,selectedtext,andselectioncolors,orchooseacustomcolorforeach
category.
ThedefaultcolorsforgeditnormallymatchthestandardGNOMEdesktopthemeselected
forthedesktop.Thesecolorswillchangetomatchtheschemeyouselectforthedesktop.
Managingplug-ins
The Plugins tab provides control over the plug-ins used in gedit. Plug-ins are separate
programsthatcaninterfacewithgedittoprovideadditionalfunctionality.
Severalplug-insareavailableforgedit,butnotallofthemareinstalledbydefault.Table
10.5describestheplug-insthatarecurrentlyavailableintheGNOMEdesktop’sgedit.
Table10.5TheGNOMEdesktopgeditPlug-ins
Plug-In
Description
Change
Changesthecaseofselectedtext
Case
Document
Reportsthenumberofwords,lines,characters,andnon-spacecharacters
Statistics
External
Providesashellenvironmentintheeditortoexecutecommandsandscripts
Tools
File
Browser
Pane
Indent
Lines
Insert
Date/Time
Modelines
Python
Console
Quick
Open
Snippets
Sort
Spell
Checker
TagList
Providesasimplefilebrowsertomakeselectingfilesforeditingeasier
Providesselectedlinestobeindentedorun-indented
Insertsthecurrentdateandtimeinseveralformatsatthecurrentcursor
position
Providesemacs-stylemessagelinesatthebottomoftheeditorwindow
Providesaninteractiveconsoleatthebottomoftheeditorwindowfor
enteringcommandsusingthePythonprogramminglanguage
Opensfilesdirectlyinthegediteditwindow
Allowsyoutostoreoften-usedpiecesoftextforeasyretrievalanywherein
thetext
Quicklysortstheentirefileorselectedtext
Providesdictionaryspellcheckingforthetextfile
Providesalistofcommonlyusedstringsyoucaneasilyenterintoyourtext
Plug-insthatareenabledshowacheckmarkinthecheckboxnexttotheirname.Some
plug-ins,suchastheExternalToolsplug-in,alsoprovideadditionalconfigurationfeatures
afteryouselectthem.Itallowsyoutosetashortcutkeytostarttheterminal,wheregedit
displaysoutput,andthecommandtousetostarttheshellsession.
Unfortunately,notallplug-insareinstalledinthesameplaceinthegeditmenubar.Some
plug-insappearintheToolsmenubaritem(suchastheSpellCheckerandExternalTools
plug-ins), while others appear in the Edit menu bar item (such as the Change Case and
InsertDate/Timeplug-ins).
ThischapterhascoveredjustafewofthetexteditorsavailableonLinux.Ifyoufindthat
thetexteditorsdescribedheredon’tmeetyourneeds,youhaveoptions.ManymoreLinux
editorsareavailable,suchasgeany,Eclipse,jed,Bluefish,andleafpadtonameafew.All
theseeditorscanhelpyouasyoubeginyourbashshellscriptwritingjourney.
Summary
Whenitcomestocreatingshellscripts,youneedsometypeoftexteditor.Severalpopular
texteditorsareavailablefortheLinuxenvironment.ThemostpopulareditorintheUnix
world,vi,hasbeenportedtotheLinuxworldasthevimeditor.Thevimeditorprovides
simpletexteditingfromtheconsole,usingarudimentaryfull-screengraphicalmode.The
vim editor provides many advanced editor features, such as text searching and
replacement.
AnothereditorthathasbeenportedfromtheUnixworldtoLinuxisthenanotexteditor.
The vim editor can be rather complex, but the nano editor offers simplicity. The nano
editorallowsquicktexteditinginconsolemode.
AnotherpopularUnixeditor—emacs—hasalsomadeitswaytotheLinuxworld.The
Linux version of emacs has both console and a graphical mode, making it the bridge
between the old world and the new. The emacs editor provides multiple buffer areas,
allowingyoutoeditmultiplefilessimultaneously.
TheKDEProjectcreatedtwoeditorsforuseintheKDEdesktop.TheKWriteeditorisa
simple editor that provides the basic text-editing features, along with a few advanced
features, such as syntax highlighting for programming code, line numbering, and code
folding. The Kate editor provides more advanced features for programmers. One great
feature in Kate is a built-in terminal window. You can open a command line interface
session directly in the Kate editor without having to open a separate terminal emulator
window. The Kate editor also allows you to open multiple files, providing different
windowsforeachopenedfile.
TheGNOMEProjectalsoprovidesasimpletexteditorforprogrammers.Thegediteditor
is a basic text editor that provides some advanced features such as code syntax
highlightingandlinenumbering,butitwasdesignedtobeabare-boneseditor.Tospruce
up the gedit editor, developers created plug-ins, which expand the features available in
gedit.Currentplug-insincludeaspell-checker,aterminalemulator,andafilebrowser.
ThiswrapsupthebackgroundchaptersonworkingwiththecommandlineinLinux.The
next part of the book dives into the shell-scripting world. The next chapter starts off by
showingyouhowtocreateashellscriptfileandhowtorunitonyourLinuxsystem.It
also shows you the basics of shell scripts, allowing you to create simple programs by
stringingmultiplecommandstogetherintoascriptyoucanrun.
PartII
ShellScriptingBasics
InThisPart
1. Chapter11BasicScriptBuilding
1. Chapter12UsingStructuredCommands
1. Chapter13MoreStructuredCommands
1. Chapter14HandlingUserInput
1. Chapter15PresentingData
1. Chapter16ScriptControl
Chapter11
BasicScriptBuilding
InThisChapter
1. Usingmultiplecommands
2. Creatingascriptfile
3. Displayingmessages
4. Usingvariables
5. Redirectinginputandoutput
6. Pipes
7. Performingmath
8. Exitingthescript
Nowthatwe’vecoveredthebasicsoftheLinuxsystemandthecommandline,it’s
timetostartcoding.Thischapterdiscussesthebasicsofwritingshellscripts.You
needtoknowthesebasicconceptsbeforeyoucanstartwritingyourownshellscript
masterpieces.
UsingMultipleCommands
So far you’ve seen how to use the command line interface (CLI) prompt of the shell to
entercommandsandviewthecommandresults.Thekeytoshellscriptsistheabilityto
enter multiple commands and process the results from each command, even possibly
passingtheresultsofonecommandtoanother.Theshellallowsyoutochaincommands
togetherintoasinglestep.
Ifyouwanttoruntwocommandstogether,youcanenterthemonthesamepromptline,
separatedwithasemicolon:
$date;who
MonFeb2115:36:09EST2014
Christinetty22014-02-2115:26
Samanthatty32014-02-2115:26
Timothytty12014-02-2115:26
usertty72014-02-1914:03(:0)
userpts/02014-02-2115:21(:0.0)
$
Congratulations,youjustwroteashellscript!Thissimplescriptusesjusttwo bashshell
commands.The datecommandrunsfirst,displayingthecurrentdateandtime,followed
by the output of the who command, showing who is currently logged on to the system.
Usingthistechnique,youcanstringtogetherasmanycommandsasyouwish,uptothe
maximumcommandlinecharactercountof255characters.
Usingthistechniqueisfineforsmallscripts,butithasamajordrawback:Youmustenter
the entire command at the command prompt every time you want to run it. Instead of
having to manually enter the commands onto a command line, you can combine the
commandsintoasimpletextfile.Whenyouneedtorunthecommands,justsimplyrun
thetextfile.
CreatingaScriptFile
Toplaceshellcommandsinatextfile,firstyouneedtouseatexteditor(seeChapter10)
tocreateafileandthenenterthecommandsintothefile.
Whencreatingashellscriptfile,youmustspecifytheshellyouareusinginthefirstline
ofthefile.Here’stheformatforthis:
#!/bin/bash
Inanormalshellscriptline,thepoundsign(#)isusedasacommentline.Acommentline
inashellscriptisn’tprocessedbytheshell.However,thefirstlineofashellscriptfileisa
special case, and the pound sign followed by the exclamation point tells the shell what
shelltorunthescriptunder(yes,youcanbeusingabashshellandrunyourscriptusing
anothershell).
Afterindicatingtheshell,commandsareenteredontoeachlineofthefile,followedbya
carriage return. As mentioned, comments can be added by using the pound sign. An
examplelookslikethis:
#!/bin/bash
#Thisscriptdisplaysthedateandwho’sloggedon
date
who
And that’s all there is to it. You can use the semicolon and put both commands on the
same line if you want to, but in a shell script, you can list commands on separate lines.
Theshellprocessescommandsintheorderinwhichtheyappearinthefile.
Alsonoticethatanotherlinewasincludedthatstartswiththepoundsymbolandaddsa
comment. Lines that start with the pound symbol (other than the first #! line) aren’t
interpretedbytheshell.Thisisagreatwaytoleavecommentsforyourselfaboutwhat’s
happening in the script, so when you come back to it two years later, you can easily
rememberwhatyoudid.
Savethisscriptinafilecalled test1,andyouarealmostready.Youneedtodoacouple
ofthingsbeforeyoucanrunyournewshellscriptfile.
Ifyoutryrunningthefilenow,you’llbesomewhatdisappointedtoseethis:
$test1
bash:test1:commandnotfound
$
Thefirsthurdletojumpisgettingthebashshelltofindyourscriptfile.Ifyouremember
fromChapter6,theshellusesanenvironmentvariablecalled PATHtofindcommands.A
quicklookatthePATHenvironmentvariabledemonstratesourproblem:
$echo$PATH
/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/bin:/usr/bin
:/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/user/bin$
The PATH environment variable is set to look for commands only in a handful of
directories.Togettheshelltofindthetest1script,weneedtodooneoftwothings:
AddthedirectorywhereourshellscriptfileislocatedtothePATHenvironment
variable.
Useanabsoluteorrelativefilepathtoreferenceourshellscriptfileintheprompt.
Tip
SomeLinuxdistributionsaddthe$HOME/bindirectorytothePATHenvironment
variable.Thiscreatesaplaceineveryuser’sHOMEdirectorytoplacefileswherethe
shellcanfindthemtoexecute.
Forthisexample,weusethesecondmethodtotelltheshellexactlywherethescriptfileis
located.Rememberthattoreferenceafileinthecurrentdirectory,youcanusethesingle
dotoperatorintheshell:
$./test1
bash:./test1:Permissiondenied
$
The shell found the shell script file just fine, but there’s another problem. The shell
indicated that you don’t have permission to execute the file. A quick look at the file
permissionsshouldshowwhat’sgoingonhere:
$ls-ltest1
-rw-rw-r—1useruser73Sep2419:56test1
$
Whenthenew test1filewascreated,the umaskvaluedeterminedthedefaultpermission
settings for the new file. Because the umask variable is set to 002 (see Chapter 7) in
Ubuntu,thesystemcreatedthefilewithonlyread/writepermissionsforthefile’sowner
andgroup.
The next step is to give the file owner permission to execute the file, using the chmod
command(seeChapter7):
$chmodu+xtest1
$./test1
MonFeb2115:38:19EST2014
Christinetty22014-02-2115:26
Samanthatty32014-02-2115:26
Timothytty12014-02-2115:26
usertty72014-02-1914:03(:0)
userpts/02014-02-2115:21(:0.0)$
Success!Nowallthepiecesareintherightplacestoexecutethenewshellscriptfile.
DisplayingMessages
Most shell commands produce their own output, which is displayed on the console
monitorwherethescriptisrunning.Manytimes,however,youwillwanttoaddyourown
textmessagestohelpthescriptuserknowwhatishappeningwithinthescript.Youcando
thiswiththe echocommand.The echocommandcandisplayasimpletextstringifyou
addthestringfollowingthecommand:
$echoThisisatest
Thisisatest
$
Notice that by default you don’t need to use quotes to delineate the string you’re
displaying. However, sometimes this can get tricky if you are using quotes within your
string:
$echoLet’sseeifthis’llwork
Letsseeifthisllwork
$
Theechocommanduseseitherdoubleorsinglequotestodelineatetextstrings.Ifyouuse
themwithinyourstring,youneedtouseonetypeofquotewithinthetextandtheother
typetodelineatethestring:
$echo“Thisisatesttoseeifyou’repayingattention”
Thisisatesttoseeifyou’repayingattention
$echo‘Richsays“scriptingiseasy”.’
Richsays“scriptingiseasy”.
$
Nowallthequotationmarksappearproperlyintheoutput.
You can add echo statements anywhere in your shell scripts where you need to display
additionalinformation:
$cattest1
#!/bin/bash
#Thisscriptdisplaysthedateandwho’sloggedon
echoThetimeanddateare:
date
echo“Let’sseewho’sloggedintothesystem:”
who
$
Whenyourunthisscript,itproducesthefollowingoutput:
$./test1
Thetimeanddateare:
MonFeb2115:41:13EST2014
Let’sseewho’sloggedintothesystem:
Christinetty22014-02-2115:26
Samanthatty32014-02-2115:26
Timothytty12014-02-2115:26
usertty72014-02-1914:03(:0)
userpts/02014-02-2115:21(:0.0)
$
That’s nice, but what if you want to echo a text string on the same line as a command
output?Youcanusethe -nparameterforthe echostatementtodothat.Justchangethe
firstechostatementlinetothis:
echo-n“Thetimeanddateare:“
Youneedtousequotesaroundthestringtoensurethatthere’saspaceattheendofthe
echoed string. The command output begins exactly where the string output stops. The
outputnowlookslikethis:
$./test1
Thetimeanddateare:MonFeb2115:42:23EST2014
Let’sseewho’sloggedintothesystem:
Christinetty22014-02-2115:26
Samanthatty32014-02-2115:26
Timothytty12014-02-2115:26
usertty72014-02-1914:03(:0)
userpts/02014-02-2115:21(:0.0)
$
Perfect! The echo command is a crucial piece of shell scripts that interact with users.
You’llfindyourselfusingitinmanysituations,especiallywhenyouwanttodisplaythe
valuesofscriptvariables.Let’slookatthatnext.
UsingVariables
Just running individual commands from the shell script is useful, but this has its
limitations.Often,you’llwanttoincorporateotherdatainyourshellcommandstoprocess
information.Youcandothisbyusingvariables.Variablesallowyoutotemporarilystore
informationwithintheshellscriptforusewithothercommandsinthescript.Thissection
showshowtousevariablesinyourshellscripts.
Environmentvariables
You’ve already seen one type of Linux variable in action. Chapter 6 described the
environment variables available in the Linux system. You can access these values from
yourshellscriptsaswell.
Theshellmaintainsenvironmentvariablesthattrackspecificsysteminformation,suchas
thenameofthesystem,thenameoftheuserloggedintothesystem,theuser’ssystemID
(calledUID),thedefaulthomedirectoryoftheuser,andthesearchpathusedbytheshell
to find programs. You can display a complete list of active environment variables
availablebyusingthesetcommand:
$set
BASH=/bin/bash
[…]
HOME=/home/Samantha
HOSTNAME=localhost.localdomain
HOSTTYPE=i386
IFS=$’\t\n’
IMSETTINGS_INTEGRATE_DESKTOP=yes
IMSETTINGS_MODULE=none
LANG=en_US.utf8
LESSOPEN=’|/usr/bin/lesspipe.sh%s’
LINES=24
LOGNAME=Samantha
[…]
You can tap into these environment variables from within your scripts by using the
environment variable’s name preceded by a dollar sign. This is demonstrated in the
followingscript:
$cattest2
#!/bin/bash
#displayuserinformationfromthesystem.
echo“Userinfoforuserid:$USER”
echoUID:$UID
echoHOME:$HOME
$
The $USER, $UID, and $HOME environment variables are used to display the pertinent
informationaboutthelogged-inuser.Theoutputshouldlooksomethinglikethis:
$chmodu+xtest2
$./test2
Userinfoforuserid:Samantha
UID:1001
HOME:/home/Samantha
$
Noticethattheenvironmentvariablesinthe echocommandsarereplacedbytheircurrent
values when the script runs. Also notice that we were able to place the $USER system
variable within the double quotation marks in the first string, and the shell script still
figuredoutwhatwemeant.Thereisadrawbacktousingthismethod,however.Lookat
whathappensinthisexample:
$echo“Thecostoftheitemis$15”
Thecostoftheitemis5
That is obviously not what was intended. Whenever the script sees a dollar sign within
quotes,itassumesyou’rereferencingavariable.Inthisexample,thescriptattemptedto
display the variable $1 (which was not defined) and then the number 5. To display an
actualdollarsign,youmustprecedeitwithabackslashcharacter:
$echo“Thecostoftheitemis\$15”
Thecostoftheitemis$15
That’sbetter.Thebackslashallowedtheshellscripttointerpretthedollarsignasanactual
dollarsignandnotavariable.Thenextsectionshowshowtocreateyourownvariablesin
yourscripts.
Note
Youmayalsoseevariablesreferencedusingtheformat${variable}.Theextrabraces
aroundthevariablenameareoftenusedtohelpidentifythevariablenamefromthe
dollarsign.
Uservariables
Inadditiontotheenvironmentvariables,ashellscriptallowsyoutosetanduseyourown
variableswithinthescript.Settingvariablesallowsyoutotemporarilystoredataanduseit
throughoutthescript,makingtheshellscriptmorelikearealcomputerprogram.
Uservariablescanbeanytextstringofupto20letters,digits,oranunderscorecharacter.
Uservariablesarecasesensitive,sothevariable Var1isdifferentfromthevariable var1.
Thislittleruleoftengetsnovicescriptprogrammersintrouble.
Valuesareassignedtouservariablesusinganequalsign.Nospacescanappearbetween
thevariable,theequalsign,andthevalue(anothertroublespotfornovices).Herearea
fewexamplesofassigningvaluestouservariables:
var1=10
var2=-57
var3=testing
var4=“stillmoretesting”
The shell script automatically determines the data type used for the variable value.
Variables defined within the shell script maintain their values throughout the life of the
shellscriptbutaredeletedwhentheshellscriptcompletes.
Justlikesystemvariables,uservariablescanbereferencedusingthedollarsign:
$cattest3
#!/bin/bash
#testingvariables
days=10
guest=“Katie”
echo“$guestcheckedin$daysdaysago”
days=5
guest=“Jessica”
echo“$guestcheckedin$daysdaysago”
$
Runningthescriptproducesthefollowingoutput:
$chmodu+xtest3
$./test3
Katiecheckedin10daysago
Jessicacheckedin5daysago
$
Each time the variable is referenced, it produces the value currently assigned to it. It’s
importanttorememberthatwhenreferencingavariablevalueyouusethedollarsign,but
whenreferencingthevariabletoassignavaluetoit,youdonotusethedollarsign.Here’s
anexampleofwhatImean:
$cattest4
#!/bin/bash
#assigningavariablevaluetoanothervariable
value1=10
value2=$value1
echoTheresultingvalueis$value2
$
Whenyouusethevalueofthevalue1variableintheassignmentstatement,youmuststill
usethedollarsign.Thiscodeproducesthefollowingoutput:
$chmodu+xtest4
$./test4
Theresultingvalueis10
$
Ifyouforgetthedollarsignandmakethevalue2assignmentlinelooklikethis:
value2=value1
yougetthefollowingoutput:
$./test4
Theresultingvalueisvalue1
$
Without the dollar sign, the shell interprets the variable name as a normal text string,
whichismostlikelynotwhatyouwanted.
Commandsubstitution
Oneofthemostusefulfeaturesofshellscriptsistheabilitytoextractinformationfrom
the output of a command and assign it to a variable. After you assign the output to a
variable, you can use that value anywhere in your script. This comes in handy when
processingdatainyourscripts.
Therearetwowaystoassigntheoutputofacommandtoavariable:
Thebacktickcharacter(`)
The$()format
Becarefulwiththebacktickcharacter;itisnotthenormalsinglequotationmarkcharacter
youareusedtousingforstrings.Becauseitisnotusedveryoftenoutsideofshellscripts,
youmaynotevenknowwheretofinditonyourkeyboard.Youshouldbecomefamiliar
withitbecauseit’sacrucialcomponentofmanyshellscripts.Hint:OnaU.S.keyboard,it
isusuallyonthesamekeyasthetildesymbol(∼).
Commandsubstitutionallowsyoutoassigntheoutputofashellcommandtoavariable.
Althoughthisdoesn’tseemlikemuch,itisamajorbuildingblockinscriptprogramming.
Youmusteithersurroundtheentirecommandlinecommandwithtwobacktickcharacters:
testing=‘date’
orusethe$()format:
testing=$(date)
Theshellrunsthecommandwithinthecommandsubstitutioncharactersandassignsthe
output to the variable testing. Notice that there are no spaces between the assignment
equal sign and the command substitution character. Here’s an example of creating a
variableusingtheoutputfromanormalshellcommand:
$cattest5
#!/bin/bash
testing=$(date)
echo“Thedateandtimeare:”$testing
$
Thevariabletestingreceivestheoutputfromthedatecommand,anditisusedintheecho
statementtodisplayit.Runningtheshellscriptproducesthefollowingoutput:
$chmodu+xtest5
$./test5
Thedateandtimeare:MonJan3120:23:25EDT2014
$
That’snotallthatexcitinginthisexample(youcouldjustaseasilyjustputthecommand
intheechostatement),butafteryoucapturethecommandoutputinavariable,youcando
anythingwithit.
Here’sapopularexampleofhowcommandsubstitutionisusedtocapturethecurrentdate
anduseittocreateauniquefilenameinascript:
#!/bin/bash
#copythe/usr/bindirectorylistingtoalogfile
today=$(date+%y%m%d)
ls/usr/bin-al>log.$today
Thetodayvariableisassignedtheoutputofaformatteddatecommand.Thisisacommon
techniqueusedtoextractdateinformationforlogfilenames.The+%y%m%dformatinstructs
thedatecommandtodisplaythedateasatwo-digityear,month,andday:
$date+%y%m%d
140131
$
Thescriptassignsthevaluetoavariable,whichisthenusedaspartofafilename.Thefile
itself contains the redirected output (discussed in the “Redirecting Input and Output”
section)ofadirectorylisting.Afterrunningthescript,youshouldseeanewfileinyour
directory:
-rw-r—r—1useruser769Jan3110:15log.140131
Thelogfileappearsinthedirectoryusingthevalueofthe $todayvariableaspartofthe
filename.Thecontentsofthelogfilearethedirectorylistingfromthe/usr/bindirectory.
Ifthescriptrunsthenextday,thelogfilenameislog.140201,thuscreatinganewfilefor
thenewday.
Caution
Commandsubstitutioncreateswhat’scalledasubshelltoruntheenclosed
command.Asubshellisaseparatechildshellgeneratedfromtheshellthat’srunning
thescript.Becauseofthat,anyvariablesyoucreateinthescriptaren’tavailableto
thesubshellcommand.
Subshellsarealsocreatedifyourunacommandfromthecommandpromptusingthe
./path,buttheyaren’tcreatedifyoujustrunthecommandwithoutapath.However,
ifyouuseabuilt-inshellcommand,thatdoesn’tgenerateasubshell.Becarefulwhen
runningscriptsfromthecommandprompt!
RedirectingInputandOutput
Sometimes, you want to save the output from a command instead of just having it
displayedonthemonitor.Thebashshellprovidesafewdifferentoperatorsthatallowyou
to redirect the output of a command to an alternative location (such as a file).
Redirection can be used for input as well as output, redirecting a file to a command for
input.Thissectiondescribeswhatyouneedtodotouseredirectioninyourshellscripts.
Outputredirection
Themostbasictypeofredirectionissendingoutputfromacommandtoafile.Thebash
shellusesthegreater-thansymbol(>)forthis:
command>outputfile
Anything that would appear on the monitor from the command instead is stored in the
outputfilespecified:
$date>test6
$ls-ltest6
-rw-r—r—1useruser29Feb1017:56test6
$cattest6
ThuFeb1017:56:58EDT2014
$
The redirect operator created the file test6 (using the default umask settings) and
redirectedtheoutputfromthedatecommandtothe test6file.Iftheoutputfilealready
exists,theredirectoperatoroverwritestheexistingfilewiththenewfiledata:
$who>test6
$cattest6
userpts/0Feb1017:55
$
Nowthecontentsofthetest6filecontaintheoutputfromthewhocommand.
Sometimes,insteadofoverwritingthefile’scontents,youmayneedtoappendoutputfrom
acommandtoanexistingfile—forexample,ifyou’recreatingalogfiletodocumentan
actiononthesystem.Inthissituation,youcanusethedoublegreater-thansymbol(>>)to
appenddata:
$date>>test6
$cattest6
userpts/0Feb1017:55
ThuFeb1018:02:14EDT2014
$
The test6filestillcontainstheoriginaldatafromthe whocommandprocessedearlier—
andnowitcontainsthenewoutputfromthedatecommand.
Inputredirection
Input redirection is the opposite of output redirection. Instead of taking the output of a
command and redirecting it to a file, input redirection takes the content of a file and
redirectsittoacommand.
Theinputredirectionsymbolistheless-thansymbol(<):
command<inputfile
Theeasywaytorememberthisisthatthecommandisalwayslistedfirstinthecommand
line, and the redirection symbol “points” to the way the data is flowing. The less-than
symbolindicatesthatthedataisflowingfromtheinputfiletothecommand.
Here’sanexampleofusinginputredirectionwiththewccommand:
$wc<test6
21160
$
Thewccommandprovidesacountoftextinthedata.Bydefault,itproducesthreevalues:
Thenumberoflinesinthetext
Thenumberofwordsinthetext
Thenumberofbytesinthetext
Byredirectingatextfiletothewccommand,youcangetaquickcountofthelines,words,
andbytesinthefile.Theexampleshowsthatthereare2lines,11words,and60bytesin
thetest6file.
Anothermethodofinputredirectioniscalledinlineinputredirection.Thismethodallows
youtospecifythedataforinputredirectiononthecommandlineinsteadofinafile.This
mayseemsomewhatoddatfirst,butafewapplicationsareavailableforthisprocess(such
asthoseshowninthe“PerformingMath”section).
The inline input redirection symbol is the double less-than symbol (<<). Besides this
symbol,youmustspecifyatextmarkerthatdelineatesthebeginningandendofthedata
usedforinput.Youcanuseanystringvalueforthetextmarker,butitmustbethesameat
thebeginningofthedataandtheendofthedata:
command<<marker
data
marker
Whenusinginlineinputredirectiononthecommandline,theshellpromptsfordatausing
the secondary prompt, defined in the PS2 environment variable (see Chapter 6). Here’s
howthislookswhenyouuseit:
$wc<<EOF
>teststring1
>teststring2
>teststring3
>EOF
3942
$
Thesecondarypromptcontinuestopromptformoredatauntilyouenterthestringvalue
forthetextmarker.The wccommandperformstheline,word,andbytecountsofthedata
suppliedbytheinlineinputredirection.
Pipes
Sometimes, you need to send the output of one command to the input of another
command.Thisispossibleusingredirection,butsomewhatclunky:
$rpm-qa>rpm.list
$sort<rpm.list
abrt-1.1.14-1.fc14.i686
abrt-addon-ccpp-1.1.14-1.fc14.i686
abrt-addon-kerneloops-1.1.14-1.fc14.i686
abrt-addon-python-1.1.14-1.fc14.i686
abrt-desktop-1.1.14-1.fc14.i686
abrt-gui-1.1.14-1.fc14.i686
abrt-libs-1.1.14-1.fc14.i686
abrt-plugin-bugzilla-1.1.14-1.fc14.i686
abrt-plugin-logger-1.1.14-1.fc14.i686
abrt-plugin-runapp-1.1.14-1.fc14.i686
acl-2.2.49-8.fc14.i686
[…]
TherpmcommandmanagesthesoftwarepackagesinstalledonsystemsusingtheRedHat
Package Management system (RPM), such as the Fedora system as shown. When used
with the -qa parameters, it produces a list of the existing packages installed, but not
necessarily in any specific order. If you’re looking for a specific package or group of
packages,itcanbedifficulttofinditusingtheoutputoftherpmcommand.
Usingthestandardoutputredirection,theoutputwasredirectedfromtherpmcommandto
afile,called rpm.list.Afterthecommandfinished,the rpm.listfilecontainedalistof
alltheinstalledsoftwarepackagesonmysystem.Next,inputredirectionwasusedtosend
the contents of the rpm.list file to the sort command to sort the package names
alphabetically.
Thatwasuseful,butagain,asomewhatclunkywayofproducingtheinformation.Instead
of redirecting the output of a command to a file, you can redirect the output to another
command.Thisprocessiscalledpiping.
Likethecommandsubstitutionbacktick,thesymbolforpipingisnotusedoftenoutsideof
shellscripting.Thesymbolistwoverticallines,oneabovetheother.However,the pipe
symboloftenlookslikeasingleverticallineinprint(|).OnaU.S.keyboard,itisusually
onthesamekeyasthebackslash(∖).The pipeisputbetweenthecommandstoredirect
theoutputfromonetotheother:
command1|command2
Don’tthinkofpipingasrunningtwocommandsbacktoback.TheLinuxsystemactually
runsbothcommandsatthesametime,linkingthemtogetherinternallyinthesystem.As
the first command produces output, it’s sent immediately to the second command. No
intermediatefilesorbufferareasareusedtotransferthedata.
Now,usingpipingyoucaneasilypipetheoutputoftherpmcommanddirectlytothesort
commandtoproduceyourresults:
$rpm-qa|sort
abrt-1.1.14-1.fc14.i686
abrt-addon-ccpp-1.1.14-1.fc14.i686
abrt-addon-kerneloops-1.1.14-1.fc14.i686
abrt-addon-python-1.1.14-1.fc14.i686
abrt-desktop-1.1.14-1.fc14.i686
abrt-gui-1.1.14-1.fc14.i686
abrt-libs-1.1.14-1.fc14.i686
abrt-plugin-bugzilla-1.1.14-1.fc14.i686
abrt-plugin-logger-1.1.14-1.fc14.i686
abrt-plugin-runapp-1.1.14-1.fc14.i686
acl-2.2.49-8.fc14.i686
[…]
Unless you’re a (very) quick reader, you probably couldn’t keep up with the output
generatedbythiscommand.Becausethepipingfeatureoperatesinrealtime,assoonas
the rpmcommandproducesdata,the sortcommandgetsbusysortingit.Bythetimethe
rpmcommandfinishesoutputtingdata,the sortcommandalreadyhasthedatasortedand
startsdisplayingitonthemonitor.
There’s no limit to the number of pipes you can use in a command. You can continue
pipingtheoutputofcommandstoothercommandstorefineyouroperation.
Inthiscase,becausetheoutputofthe sortcommandzoomsbysoquickly,youcanuse
one of the text paging commands (such as less or more) to force the output to stop at
everyscreenofdata:
$rpm-qa|sort|more
Thiscommandsequencerunsthe rpmcommand, pipestheoutputtothe sortcommand,
andthen pipesthatoutputtothe morecommandtodisplaythedata,stoppingafterevery
screen of information. This now lets you pause and read what’s on the display before
continuing,asshowninFigure11.1.
Figure11.1Usingpipingtosenddatatothemorecommand
Togetevenfancier,youcanuseredirectionalongwithpipingtosaveyouroutputtoafile:
$rpm-qa|sort>rpm.list
$morerpm.list
abrt-1.1.14-1.fc14.i686
abrt-addon-ccpp-1.1.14-1.fc14.i686
abrt-addon-kerneloops-1.1.14-1.fc14.i686
abrt-addon-python-1.1.14-1.fc14.i686
abrt-desktop-1.1.14-1.fc14.i686
abrt-gui-1.1.14-1.fc14.i686
abrt-libs-1.1.14-1.fc14.i686
abrt-plugin-bugzilla-1.1.14-1.fc14.i686
abrt-plugin-logger-1.1.14-1.fc14.i686
abrt-plugin-runapp-1.1.14-1.fc14.i686
acl-2.2.49-8.fc14.i686
[…]
Asexpected,thedataintherpm.listfileisnowsorted!
By far one of the most popular uses of piping is piping the results of commands that
produce long output to the more command. This is especially common with the ls
command,asshowninFigure11.2.
Figure11.2Usingthemorecommandwiththelscommand
Thels-lcommandproducesalonglistingofallthefilesinthedirectory.Fordirectories
withlotsoffiles,thiscanbequitealisting.Bypipingtheoutputtothe morecommand,
youforcetheoutputtostopattheendofeveryscreenofdata.
PerformingMath
Another feature crucial to any programming language is the ability to manipulate
numbers. Unfortunately, for shell scripts this process is a bit awkward. There are two
differentwaystoperformmathematicaloperationsinyourshellscripts.
Theexprcommand
Originally, the Bourne shell provided a special command that was used for processing
mathematicalequations.Theexprcommandallowedtheprocessingofequationsfromthe
commandline,butitisextremelyclunky:
$expr1+5
6
The exprcommandrecognizesafewdifferentmathematicalandstringoperators,shown
inTable11.1.
Table11.1TheexprCommandOperators
Operator
ARG1|ARG2
Description
ReturnsARG1ifneitherargumentisnullorzero;otherwise,returns
ARG2
STRING:REGEXP
ReturnsARG1ifneitherargumentisnullorzero;otherwise,returns
0
Returns1ifARG1islessthanARG2;otherwise,returns0
Returns1ifARG1islessthanorequaltoARG2;otherwise,returns0
Returns1ifARG1isequaltoARG2;otherwise,returns0
Returns1ifARG1isnotequaltoARG2;otherwise,returns0
Returns1ifARG1isgreaterthanorequaltoARG2;otherwise,
returns0
Returns1ifARG1isgreaterthanARG2;otherwise,returns0
ReturnsthearithmeticsumofARG1andARG2
ReturnsthearithmeticdifferenceofARG1andARG2
ReturnsthearithmeticproductofARG1andARG2
ReturnsthearithmeticquotientofARG1dividedbyARG2
ReturnsthearithmeticremainderofARG1dividedbyARG2
ReturnsthepatternmatchifREGEXPmatchesapatterninSTRING
matchSTRING
REGEXP
ReturnsthepatternmatchifREGEXPmatchesapatterninSTRING
substrSTRINGPOS
LENGTH
ReturnsthesubstringLENGTHcharactersinlength,startingat
positionPOS(startingat1)
ReturnspositioninSTRINGwhereCHARSisfound;otherwise,
returns0
ARG1&ARG2
ARG1<ARG2
ARG1<=ARG2
ARG1=ARG2
ARG1!=ARG2
ARG1>=ARG2
ARG1>ARG2
ARG1+ARG2
ARG1-ARG2
ARG1*ARG2
ARG1/ARG2
ARG1%ARG2
indexSTRINGCHARS
lengthSTRING
+TOKEN
(EXPRESSION)
ReturnsthenumericlengthofthestringSTRING
InterpretsTOKENasastring,evenifit’sakeyword
ReturnsthevalueofEXPRESSION
Although the standard operators work fine in the expr command, the problem occurs
when using them from a script or the command line. Many of the expr command
operatorshaveothermeaningsintheshell(suchastheasterisk).Usingtheminthe expr
commandproducesoddresults:
$expr5*2
expr:syntaxerror
$
To solve this problem, you need to use the shell escape character (the backslash) to
identifyanycharactersthatmaybemisinterpretedbytheshellbeforebeingpassedtothe
exprcommand:
$expr5\*2
10
$
Nowthat’sreallystartingtogetugly!Usingtheexprcommandinashellscriptisequally
cumbersome:
$cattest6
#!/bin/bash
#Anexampleofusingtheexprcommand
var1=10
var2=20
var3=$(expr$var2/$var1)
echoTheresultis$var3
Toassigntheresultofamathematicalequationtoavariable,youhavetousecommand
substitutiontoextracttheoutputfromtheexprcommand:
$chmodu+xtest6
$./test6
Theresultis2
$
Fortunately,thebashshellhasanimprovementforprocessingmathematicaloperatorsas
youshallseeinthenextsection.
Usingbrackets
The bash shell includes the expr command to stay compatible with the Bourne shell;
however, it also provides a much easier way of performing mathematical equations. In
bash, when assigning a mathematical value to a variable, you can enclose the
mathematicalequationusingadollarsignandsquarebrackets($[operation]):
$var1=$[1+5]
$echo$var1
6
$var2=$[$var1*2]
$echo$var2
12
$
Using brackets makes shell math much easier than with the expr command. This same
techniquealsoworksinshellscripts:
$cattest7
#!/bin/bash
var1=100
var2=50
var3=45
var4=$[$var1*($var2-$var3)]
echoThefinalresultis$var4
$
Runningthisscriptproducestheoutput:
$chmodu+xtest7
$./test7
Thefinalresultis500
$
Also, notice that when using the square brackets method for calculating equations, you
don’t need to worry about the multiplication symbol, or any other characters, being
misinterpretedbytheshell.Theshellknowsthatit’snotawildcardcharacterbecauseitis
withinthesquarebrackets.
There’s one major limitation to performing math in the bash shell script. Look at this
example:
$cattest8
#!/bin/bash
var1=100
var2=45
var3=$[$var1/$var2]
echoThefinalresultis$var3
$
Nowrunitandseewhathappens:
$chmodu+xtest8
$./test8
Thefinalresultis2
$
The bash shell mathematical operators support only integer arithmetic. This is a huge
limitationifyou’retryingtodoanysortofreal-worldmathematicalcalculations.
Note
Thezshell(zsh)providesfullfloating-pointarithmeticoperations.Ifyourequire
floating-pointcalculationsinyourshellscripts,youmightconsidercheckingoutthe
zshell(discussedinChapter23).
Afloating-pointsolution
You can use several solutions for overcoming the bash integer limitation. The most
popularsolutionusesthebuilt-inbashcalculator,calledbc.
Thebasicsofbc
Thebashcalculatorisactuallyaprogramminglanguagethatallowsyoutoenterfloatingpointexpressionsatacommandlineandtheninterpretstheexpressions,calculatesthem,
andreturnstheresult.Thebashcalculatorrecognizesthese:
Numbers(bothintegerandfloatingpoint)
Variables(bothsimplevariablesandarrays)
Comments(linesstartingwithapoundsignortheClanguage/**/pair)
Expressions
Programmingstatements(suchasif-thenstatements)
Functions
Youcanaccessthebashcalculatorfromtheshellpromptusingthebccommand:
$bc
bc1.06.95
Copyright1991-1994,1997,1998,2000,2004,2006FreeSoftwareFoundation,
Inc.
ThisisfreesoftwarewithABSOLUTELYNOWARRANTY.
Fordetailstype‘warranty’.
12*5.4
64.8
3.156*(3+5)
25.248
quit
$
Theexamplestartsoutbyenteringtheexpression 12*5.4.Thebashcalculatorreturns
the answer. Each subsequent expression entered into the calculator is evaluated, and the
resultisdisplayed.Toexitthebashcalculator,youmustenterquit.
The floating-point arithmetic is controlled by a built-in variable called scale.Youmust
setthisvaluetothedesirednumberofdecimalplacesyouwantinyouranswers,oryou
won’tgetwhatyouwerelookingfor:
$bc-q
3.44/5
0
scale=4
3.44/5
.6880
quit
$
Thedefaultvalueforthe scalevariableiszero.Beforethe scalevalueisset,thebash
calculator provides the answer to zero decimal places. After you set the scalevariable
value to four, the bash calculator displays the answer to four decimal places. The -q
commandlineparametersuppressesthelengthywelcomebannerfromthebashcalculator.
Inadditiontonormalnumbers,thebashcalculatoralsounderstandsvariables:
$bc-q
var1=10
var1*4
40
var2=var1/5
printvar2
2
quit
$
Afteravariablevalueisdefined,youcanusethevariablethroughoutthebashcalculator
session.Theprintstatementallowsyoutoprintvariablesandnumbers.
Usingbcinscripts
Nowyoumaybewonderinghowthebashcalculatorisgoingtohelpyouwithfloatingpointarithmeticinyourshellscripts.Doyouremembercommandsubstitution?Yes,you
canusethecommandsubstitutioncharactertorunabccommandandassigntheoutputto
avariable!Thebasicformattouseisthis:
variable=$(echo“options;expression”|bc)
Thefirstportion, options,allowsyoutosetvariables.Ifyouneedtosetmorethanone
variable, separate them using the semicolon. The expression parameter defines the
mathematicalexpressiontoevaluateusing bc.Here’saquickexampleofdoingthisina
script:
$cattest9
#!/bin/bash
var1=$(echo“scale=4;3.44/5”|bc)
echoTheansweris$var1
$
Thisexamplesetsthescalevariabletofourdecimalplacesandthenspecifiesaspecific
calculationfortheexpression.Runningthisscriptproducesthefollowingoutput:
$chmodu+xtest9
$./test9
Theansweris.6880
$
Nowthat’sfancy!Youaren’tlimitedtojustusingnumbersfortheexpressionvalue.You
canalsousevariablesdefinedintheshellscript:
$cattest10
#!/bin/bash
var1=100
var2=45
var3=$(echo“scale=4;$var1/$var2”|bc)
echoTheanswerforthisis$var3
$
The script defines two variables, which are used within the expression sent to the bc
command.Remembertousethedollarsigntosignifythevalueforthevariablesandnot
thevariablesthemselves.Theoutputofthisscriptisasfollows:
$./test10
Theanswerforthisis2.2222
$
And of course, after a value is assigned to a variable, that variable can be used in yet
anothercalculation:
$cattest11
#!/bin/bash
var1=20
var2=3.14159
var3=$(echo“scale=4;$var1*$var1”|bc)
var4=$(echo“scale=4;$var3*$var2”|bc)
echoThefinalresultis$var4
$
This method works fine for short calculations, but sometimes you need to get more
involvedwithyournumbers.Ifyouhavemorethanjustacoupleofcalculations,itgets
confusingtryingtolistmultipleexpressionsonthesamecommandline.
There’sasolutiontothisproblem.Thebccommandrecognizesinputredirection,allowing
you to redirect a file to the bc command for processing. However, this also can get
confusing,becauseyou’dneedtostoreyourexpressionsinafile.
The best method is to use inline input redirection, which allows you to redirect data
directlyfromthecommandline.Intheshellscript,youassigntheoutputtoavariable:
variable=$(bc<<EOF
options
statements
expressions
EOF
)
The EOF text string indicates the beginning and end of the inline redirection data.
Rememberthatthecommandsubstitutioncharactersarestillneededtoassigntheoutput
ofthebccommandtothevariable.
Now you can place all the individual bash calculator elements on separate lines in the
scriptfile.Here’sanexampleofusingthistechniqueinascript:
$cattest12
#!/bin/bash
var1=10.46
var2=43.67
var3=33.2
var4=71
var5=$(bc<<EOF
scale=4
a1=($var1*$var2)
b1=($var3*$var4)
a1+b1
EOF
)
echoThefinalanswerforthismessis$var5
$
Placingeachoptionandexpressiononaseparatelineinyourscriptmakesthingscleaner
and easier to read and follow. The EOF string indicates the start and end of the data to
redirecttothebccommand.Ofcourse,youmustusethecommandsubstitutioncharacters
toindicatethecommandtoassigntothevariable.
You’llalsonoticeinthisexamplethatyoucanassignvariableswithinthebashcalculator.
It’simportanttorememberthatanyvariablescreatedwithinthebashcalculatorarevalid
onlywithinthebashcalculatorandcan’tbeusedintheshellscript.
ExitingtheScript
Sofarinoursamplescripts,weterminatedthingsprettyabruptly.Whenwewerefinished
with our last command, we just ended the script. There’s a more elegant way of
completingthingsavailabletous.
Everycommandthatrunsintheshellusesanexitstatustoindicatetotheshellthatit’s
finishedprocessing.Theexitstatusisanintegervaluebetween0and255that’spassedby
thecommandtotheshellwhenthecommandfinishesrunning.Youcancapturethisvalue
anduseitinyourscripts.
Checkingtheexitstatus
Linux provides the $? special variable that holds the exit status value from the last
command that executed. You must view or use the $? variable immediately after the
command you want to check. It changes values to the exit status of the last command
executedbytheshell:
$date
SatJan1510:01:30EDT2014
$echo$?
0
$
By convention, the exit status of a command that successfully completes is zero. If a
commandcompleteswithanerror,thenapositiveintegervalueisplacedintheexitstatus:
$asdfg
-bash:asdfg:commandnotfound
$echo$?
127
$
The invalid command returns an exit status of 127. There’s not much of a standard
conventiontoLinuxerrorexitstatuscodes.However,youcanusetheguidelinesshownin
Table11.2.
Table11.2LinuxExitStatusCodes
Code
Description
0 Successfulcompletionofthecommand
1
Generalunknownerror
2
Misuseofshellcommand
126
Thecommandcan’texecute
127
Commandnotfound
128
Invalidexitargument
128+x
FatalerrorwithLinuxsignalx
130
CommandterminatedwithCtrl+C
255
Exitstatusoutofrange
Anexitstatusvalueof126indicatesthattheuserdidn’thavetheproperpermissionssetto
executethecommand:
$./myprog.c
-bash:./myprog.c:Permissiondenied
$echo$?
126
$
Another common error you’ll encounter occurs if you supply an invalid parameter to a
command:
$date%t
date:invaliddate‘%t’
$echo$?
1
$
Thisgeneratesthegeneralexitstatuscodeof1,indicatingthatanunknownerroroccurred
inthecommand.
Theexitcommand
Bydefault,yourshellscriptexitswiththeexitstatusofthelastcommandinyourscript:
$./test6
Theresultis2
$echo$?
0
$
Youcanchangethattoreturnyourownexitstatuscode.Theexitcommandallowsyouto
specifyanexitstatuswhenyourscriptends:
$cattest13
#!/bin/bash
#testingtheexitstatus
var1=10
var2=30
var3=$[$var1+$var2]
echoTheansweris$var3
exit5
$
Whenyouchecktheexitstatusofthescript,yougetthevalueusedastheparameterofthe
exitcommand:
$chmodu+xtest13
$./test13
Theansweris40
$echo$?
5
$
Youcanalsousevariablesintheexitcommandparameter:
$cattest14
#!/bin/bash
#testingtheexitstatus
var1=10
var2=30
var3=$[$var1+$var2]
exit$var3
$
Whenyourunthiscommand,itproducesthefollowingexitstatus:
$chmodu+xtest14
$./test14
$echo$?
40
$
Youshouldbecarefulwiththisfeature,however,becausetheexitstatuscodescanonlygo
upto255.Watchwhathappensinthisexample:
$cattest14b
#!/bin/bash
#testingtheexitstatus
var1=10
var2=30
var3=$[$var1*$var2]
echoThevalueis$var3
exit$var3
$
Nowwhenyourunit,yougetthefollowing:
$./test14b
Thevalueis300
$echo$?
44
$
Theexitstatuscodeisreducedtofitinthe0to255range.Theshelldoesthisbyusing
moduloarithmetic.The moduloofavalueistheremainderafteradivision.Theresulting
numberistheremainderofthespecifiednumberdividedby256.Inthecaseof300(the
resultvalue),theremainderis44,whichiswhatappearsastheexitstatuscode.
InChapter12,you’llseehowyoucanusethe if-thenstatementtochecktheerrorstatus
returnedbyacommandtoseewhetherthecommandwassuccessful.
Summary
Thebashshellscriptallowsyoutostringcommandstogetherintoascript.Themostbasic
way to create a script is to separate multiple commands on the command line using a
semicolon. The shell executes each command in order, displaying the output of each
commandonthemonitor.
Youcanalsocreateashellscriptfile,placingmultiplecommandsinthefilefortheshell
toexecuteinorder.Theshellscriptfilemustdefinetheshellusedtorunthescript.Thisis
doneinthefirstlineofthescriptfile,usingthe#!symbol,followedbythefullpathofthe
shell.
Within the shell script you can reference environment variable values by using a dollar
sign in front of the variable. You can also define your own variables for use within the
script, and assign values and even the output of a command by using the backtick
characterorthe$()format.Thevariablevaluecanbeusedwithinthescriptbyplacinga
dollarsigninfrontofthevariablename.
The bash shell allows you to redirect both the input and output of a command from the
standardbehavior.Youcanredirecttheoutputofanycommandfromthemonitordisplay
toafilebyusingthegreater-thansymbol,followedbythenameofthefiletocapturethe
output.Youcanappendoutputdatatoanexistingfilebyusingtwogreater-thansymbols.
Theless-thansymbolisusedtoredirectinputtoacommand.Youcanredirectinputfroma
filetoacommand.
TheLinux pipecommand(thebrokenbarsymbol)allowsyoutoredirecttheoutputofa
command directly to the input of another command. The Linux system runs both
commandsatthesametime,sendingtheoutputofthefirstcommandtotheinputofthe
secondcommandwithoutusinganyredirectfiles.
Thebashshellprovidesacoupleofwaysforyoutoperformmathematicaloperationsin
yourshellscripts.Theexprcommandisasimplewaytoperformintegermath.Inthebash
shell, you can also perform basic math calculations by enclosing equations in square
brackets, preceded by a dollar sign. To perform floating-point arithmetic, you need to
utilizethebccalculatorcommand,redirectinginputfrominlinedataandstoringtheoutput
inauservariable.
Finally, the chapter discussed how to use the exit status in your shell script. Every
commandthatrunsintheshellproducesanexitstatus.Theexitstatusisanintegervalue
between0and255thatindicatesifthecommandcompletedsuccessfully,andifnot,what
the reason may have been. An exit status of 0 indicates that the command completed
successfully.Youcanusethe exitcommandinyourshellscripttodeclareaspecificexit
statusuponthecompletionofyourscript.
So far in your shell scripts, things have proceeded in an orderly fashion from one
command to the next. In the next chapter, you’ll see how you can use some logic flow
controltoalterwhichcommandsareexecutedwithinthescript.
Chapter12
UsingStructuredCommands
InThisChapter
1. Workingwiththeif-thenstatement
2. Nestingifs
3. Understandingthetestcommand
4. Testingcompoundconditions
5. Usingdoublebracketsandparentheses
6. Lookingatcase
IntheshellscriptspresentedinChapter11,theshellprocessedeachindividual
commandintheshellscriptintheorderitappeared.Thisworksoutfinefor
sequentialoperations,whereyouwantallthecommandstoprocessintheproper
order.However,thisisn’thowallprogramsoperate.
Manyprogramsrequiresomesortoflogicflowcontrolbetweenthecommandsinthe
script.Thereisawholecommandclassthatallowsthescripttoskipoverexecuted
commandsbasedontestedconditions.Thesecommandsaregenerallyreferredtoas
structuredcommands.
Thestructuredcommandsallowyoutoaltertheoperationflowofaprogram.Quitea
fewstructuredcommandsareavailableinthebashshell,sowe’lllookatthem
individually.Inthischapter,welookatif-thenandcasestatements.
Workingwiththeif-thenStatement
The most basic type of structured command is the if-then statement. The if-then
statementhasthefollowingformat:
ifcommand
then
commands
fi
Ifyou’reusing if-thenstatementsinotherprogramminglanguages,thisformatmaybe
somewhatconfusing.Inotherprogramminglanguages,theobjectaftertheifstatementis
anequationthatisevaluatedfora TRUEor FALSEvalue.That’snothowthebashshell if
statementworks.
Thebashshell ifstatementrunsthecommanddefinedonthe ifline.Iftheexitstatusof
the command (see Chapter 11) is zero (the command completed successfully), the
commandslistedunderthethensectionareexecuted.Iftheexitstatusofthecommandis
anything else, the then commands aren’t executed, and the bash shell moves on to the
nextcommandinthescript.Thefistatementdelineatestheif-thenstatement’send.
Here’sasimpleexampletodemonstratethisconcept:
$cattest1.sh
#!/bin/bash
#testingtheifstatement
ifpwd
then
echo“Itworked”
fi
$
Thisscriptusesthe pwdcommandonthe ifline.Ifthecommandcompletessuccessfully,
the echo statement should display the text string. When you run this script from the
commandline,yougetthefollowingresults:
$./test1.sh
/home/Christine
Itworked
$
The shell executed the pwd command listed on the if line. Because the exit status was
zero,italsoexecutedtheechostatementlistedinthethensection.
Here’sanotherexample:
$cattest2.sh
#!/bin/bash
#testingabadcommand
ifIamNotaCommand
then
echo“Itworked”
fi
echo“Weareoutsidetheifstatement”
$
$./test2.sh
./test2.sh:line3:IamNotaCommand:commandnotfound
Weareoutsidetheifstatement
$
Inthisexample,wedeliberatelyusedacommand, IamNotaCommand,thatdoesnotworkin
theifstatementline.Becausethisisabadcommand,itproducesanexitstatusthat’snonzero,andthebashshellskipsthe echostatementinthe thensection.Alsonoticethatthe
errormessagegeneratedfromrunningthecommandintheifstatementstillappearsinthe
script’s output. There may be times when you don’t want an error statement to appear.
Chapter15discusseshowthiscanbeavoided.
Note
Youmightseeanalternativeformoftheif-thenstatementusedinsomescripts:
ifcommand;then
commands
fi
Byputtingasemicolonattheendofthecommandtoevaluate,youcanincludethe
thenstatementonthesameline,whichlooksclosertohowif-thenstatementsare
handledinsomeotherprogramminglanguages.
Youarenotlimitedtojustonecommandinthe thensection.Youcanlistcommandsjust
asintherestoftheshellscript.Thebashshelltreatsthecommandsasablock,executing
all of them when the command in the if statement line returns a zero exit status or
skippingallofthemwhenthecommandreturnsanon-zeroexitstatus:
$cattest3.sh
#!/bin/bash
#testingmultiplecommandsinthethensection
#
testuser=Christine
#
ifgrep$testuser/etc/passwd
then
echo“Thisismyfirstcommand”
echo“Thisismysecondcommand”
echo“Icanevenputinothercommandsbesidesecho:”
ls-a/home/$testuser/.b*
fi
$
The ifstatementlineusesthe grepcommenttosearchthe /etc/passwd file to see if a
specificusernameiscurrentlyusedonthesystem.Ifthere’sauserwiththatlogonname,
thescriptdisplayssometextandthenliststhebashfilesintheuser’sHOMEdirectory:
$./test3.sh
Christine:x:501:501:ChristineB:/home/Christine:/bin/bash
Thisismyfirstcommand
Thisismysecondcommand
Icanevenputinothercommandsbesidesecho:
/home/Christine/.bash_history/home/Christine/.bash_profile
/home/Christine/.bash_logout/home/Christine/.bashrc
$
However, if you set the testuser variable to a user that doesn’t exist on the system,
nothinghappens:
$cattest3.sh
#!/bin/bash
#testingmultiplecommandsinthethensection
#
testuser=NoSuchUser
#
ifgrep$testuser/etc/passwd
then
echo“Thisismyfirstcommand”
echo“Thisismysecondcommand”
echo“Icanevenputinothercommandsbesidesecho:”
ls-a/home/$testuser/.b*
fi
$
$./test3.sh
$
It’snotallthatexciting.Itwouldbeniceifwecoulddisplayalittlemessagesayingthat
theusernamewasn’tfoundonthesystem.Well,wecan,usinganotherfeatureofthe ifthenstatement.
Exploringtheif-then-elseStatement
Intheif-thenstatement,youhaveonlyoneoptionforwhetheracommandissuccessful.
If the command returns a non-zero exit status code, the bash shell just moves on to the
next command in the script. In this situation, it would be nice to be able to execute an
alternatesetofcommands.That’sexactlywhattheif-then-elsestatementisfor.
Theif-then-elsestatementprovidesanothergroupofcommandsinthestatement:
ifcommand
then
commands
else
commands
fi
When the command in the if statement line returns with a zero exit status code, the
commandslistedinthe thensectionareexecuted,justasinanormal if-thenstatement.
Whenthecommandinthe ifstatementlinereturnsanon-zeroexitstatuscode,thebash
shellexecutesthecommandsintheelsesection.
Nowyoucancopyandmodifythetestscripttoincludeanelsesection:
$cptest3.shtest4.sh
$
$nanotest4.sh
$
$cattest4.sh
#!/bin/bash
#testingtheelsesection
#
testuser=NoSuchUser
#
ifgrep$testuser/etc/passwd
then
echo“Thebashfilesforuser$testuserare:”
ls-a/home/$testuser/.b*
echo
else
echo“Theuser$testuserdoesnotexistonthissystem.”
echo
fi
$
$./test4.sh
TheuserNoSuchUserdoesnotexistonthissystem.
$
That’smoreuser-friendly.Justlikethethensection,theelsesectioncancontainmultiple
commands.Thefistatementdelineatestheendoftheelsesection.
Nestingifs
Sometimes,youmustcheckforseveralsituationsinyourscriptcode.Forthesesituations,
youcannesttheif-thenstatements:
Tocheckifalogonnameisnotinthe /etc/passwdfileandyetadirectoryforthatuser
stillexists,useanested if-thenstatement.Inthiscase,thenested if-thenstatementis
withintheprimaryif-then-elsestatement’selsecodeblock:
$ls-d/home/NoSuchUser/
/home/NoSuchUser/
$
$cattest5.sh
#!/bin/bash
#Testingnestedifs
#
testuser=NoSuchUser
#
ifgrep$testuser/etc/passwd
then
echo“Theuser$testuserexistsonthissystem.”
else
echo“Theuser$testuserdoesnotexistonthissystem.”
ifls-d/home/$testuser/
then
echo“However,$testuserhasadirectory.”
fi
fi
$
$./test5.sh
TheuserNoSuchUserdoesnotexistonthissystem.
/home/NoSuchUser/
However,NoSuchUserhasadirectory.
$
The script correctly finds that although the login name has been removed from the
/etc/passwdfile,theuser’sdirectoryisstillonthesystem.Theproblemwithusingthis
mannerofnested if-thenstatementsinascriptisthatthecodecangethardtoread,and
thelogicflowbecomesdifficulttofollow.
Insteadofhavingtowriteseparateif-thenstatements,youcanuseanalternativeversion
oftheelsesection,called elif.Theelifcontinuesanelsesectionwithanother if-then
statement:
ifcommand1
then
commands
elifcommand2
then
morecommands
fi
The elifstatementlineprovidesanothercommandtoevaluate,similartotheoriginal if
statementline.Iftheexitstatuscodefromthe elifcommandiszero,bashexecutesthe
commands in the second then statement section. Using this method of nesting provides
cleanercodewithaneasier-to-followlogicflow:
$cattest5.sh
#!/bin/bash
#Testingnestedifs-useelif
#
testuser=NoSuchUser
#
ifgrep$testuser/etc/passwd
then
echo“Theuser$testuserexistsonthissystem.”
#
elifls-d/home/$testuser
then
echo“Theuser$testuserdoesnotexistonthissystem.”
echo“However,$testuserhasadirectory.”
#
fi
$
$./test5.sh
/home/NoSuchUser
TheuserNoSuchUserdoesnotexistonthissystem.
However,NoSuchUserhasadirectory.
$
Youcaneventakethisscriptastepfurtherandhaveitcheckforbothanon-existentuser
with a directory and a non-existent user without a directory. This is accomplished by
addinganelsestatementwithinthenestedelif:
$cattest5.sh
#!/bin/bash
#Testingnestedifs-useelif&else
#
testuser=NoSuchUser
#
ifgrep$testuser/etc/passwd
then
echo“Theuser$testuserexistsonthissystem.”
#
elifls-d/home/$testuser
then
echo“Theuser$testuserdoesnotexistonthissystem.”
echo“However,$testuserhasadirectory.”
#
else
echo“Theuser$testuserdoesnotexistonthissystem.”
echo“And,$testuserdoesnothaveadirectory.”
fi
$
$./test5.sh
/home/NoSuchUser
TheuserNoSuchUserdoesnotexistonthissystem.
However,NoSuchUserhasadirectory.
$
$sudormdir/home/NoSuchUser
[sudo]passwordforChristine:
$
$./test5.sh
ls:cannotaccess/home/NoSuchUser:Nosuchfileordirectory
TheuserNoSuchUserdoesnotexistonthissystem.
And,NoSuchUserdoesnothaveadirectory.
$
Beforethe/home/NoSuchUserdirectorywasremovedandthetestscriptexecutedtheelif
statement, a zero exit status was returned. Thus, the statements within the elif‘s then
code block were executed. After the /home/NoSuchUser directory was removed, a nonzero exit status was returned for the elif statement. This caused the statements in the
elseblockwithintheelifblocktobeexecuted.
Tip
Keepinmindthat,withanelifstatement,anyelsestatementsimmediately
followingitareforthatelifcodeblock.Theyarenotpartofaprecedingif-then
statementcodeblock.
You can continue to string elif statements together, creating one huge if-then-elif
conglomeration:
ifcommand1
then
commandset1
elifcommand2
then
commandset2
elifcommand3
then
commandset3
elifcommand4
then
commandset4
fi
Eachblockofcommandsisexecuteddependingonwhichcommandreturnsthezeroexit
statuscode.Rememberthatthebashshellexecutesthe if statements in order, and only
thefirstonethatreturnsazeroexitstatusresultsinthethensectionbeingexecuted.
Even though the code looks cleaner with elif statements, it still can be confusing to
followthescript’slogic.Laterinthe“ConsideringthecaseCommand”section,you’llsee
howtousethecasecommandinsteadofhavingtonestlotsofif-thenstatements.
TryingthetestCommand
Sofar,allyou’veseeninthe ifstatementlinearenormalshellcommands.Youmightbe
wondering if the bash if-then statement has the ability to evaluate any condition other
thanacommand’sexitstatuscode.
Theanswerisno,itcan’t.However,there’saneatutilityavailableinthebashshellthat
helpsyouevaluateotherthings,usingtheif-thenstatement.
Thetestcommandprovidesawaytotestdifferentconditionsinanif-thenstatement.If
theconditionlistedinthetestcommandevaluatestoTRUE,thetestcommandexitswith
azeroexitstatuscode.Thismakesthe if-thenstatementbehaveinmuchthesameway
thatif-thenstatementsworkinotherprogramminglanguages.IftheconditionisFALSE,
the test command exits with a non-zero exit status code, which causes the if-then
statementtoexit.
Theformatofthetestcommandisprettysimple:
testcondition
The condition is a series of parameters and values that the test command evaluates.
Whenusedinanif-thenstatement,thetestcommandlookslikethis:
iftestcondition
then
commands
fi
If you leave out the condition portion of the test command statement, it exits with a
non-zeroexitstatuscodeandtriggersanyelseblockstatements:
$cattest6.sh
#!/bin/bash
#Testingthetestcommand
#
iftest
then
echo“NoexpressionreturnsaTrue”
else
echo“NoexpressionreturnsaFalse”
fi
$
$./test6.sh
NoexpressionreturnsaFalse
$
Whenyouaddinacondition,itistestedbythe testcommand.Forexample,usingthe
test command, you can determine whether a variable has content. A simple condition
expressionisneededtodeterminewhetheravariablehascontent:
$cattest6.sh
#!/bin/bash
#Testingthetestcommand
#
my_variable=“Full”
#
iftest$my_variable
then
echo“The$my_variableexpressionreturnsaTrue”
#
else
echo“The$my_variableexpressionreturnsaFalse”
fi
$
$./test6.sh
TheFullexpressionreturnsaTrue
$
Thevariablemy_variablecontainscontent(Full),sowhenthetestcommandchecksthe
condition,theexitstatusreturnsazero.Thistriggersthestatementinthethencodeblock.
Asyouwouldsuspect,theoppositeoccurswhenthevariabledoesnotcontaincontent:
$cattest6.sh
#!/bin/bash
#Testingthetestcommand
#
my_variable=””
#
iftest$my_variable
then
echo“The$my_variableexpressionreturnsaTrue”
#
else
echo“The$my_variableexpressionreturnsaFalse”
fi
$
$./test6.sh
TheexpressionreturnsaFalse
$
The bash shell provides an alternative way of testing a condition without declaring the
testcommandinanif-thenstatement:
if[condition]
then
commands
fi
Thesquarebracketsdefinethetestcondition.Becareful;youmusthaveaspaceafterthe
firstbracketandaspacebeforethelastbracket,oryou’llgetanerrormessage.
Thetestcommandandtestconditionscanevaluatethreeclassesofconditions:
Numericcomparisons
Stringcomparisons
Filecomparisons
The next sections describe how to use each of these test classes in your if-then
statements.
Usingnumericcomparisons
The most common test evaluation method is to perform a comparison of two numeric
values.Table12.1showsthelistofconditionparametersusedfortestingtwovalues.
Table12.1ThetestNumericComparisons
Comparison
n1-eqn2
n1-gen2
n1-gtn2
n1-len2
n1-ltn2
n1-nen2
Description
Checksifn1isequalton2
Checksifn1isgreaterthanorequalton2
Checksifn1isgreaterthann2
Checksifn1islessthanorequalton2
Checksifn1islessthann2
Checksifn1isnotequalton2
Thenumerictestconditionscanbeusedtoevaluatebothnumbersandvariables.Here’san
exampleofdoingthat:
$catnumeric_test.sh
#!/bin/bash
#Usingnumerictestevaluations
#
value1=10
value2=11
#
if[$value1-gt5]
then
echo“Thetestvalue$value1isgreaterthan5”
fi
#
if[$value1-eq$value2]
then
echo“Thevaluesareequal”
else
echo“Thevaluesaredifferent”
fi
#
$
Thefirsttestcondition:
if[$value1-gt5]
testsifthevalueofthevariablevalue1isgreaterthan5.Thesecondtestcondition:
if[$value1-eq$value2]
testsifthevalueofthevariablevalue1isequaltothevalueofthevariablevalue2.Both
numerictestconditionsevaluateasexpected:
$./numeric_test.sh
Thetestvalue10isgreaterthan5
Thevaluesaredifferent
$
Thereisalimitationtothetestnumericconditionsconcerningfloating-pointvalues:
$catfloating_point_test.sh
#!/bin/bash
#Usingfloatingpointnumbersintestevaluations
#
value1=5.555
#
echo“Thetestvalueis$value1”
#
if[$value1-gt5]
then
echo“Thetestvalue$value1isgreaterthan5”
fi
#
$./floating_point_test.sh
Thetestvalueis5.555
./floating_point_test.sh:line8:
[:5.555:integerexpressionexpected
$
Thisexampleusesafloating-pointvalue,storedinthevalue1variable.Next,itevaluates
thevalue.Somethingobviouslywentwrong.
Remember that the only numbers the bash shell can handle are integers. This works
perfectlyfineifallyouneedtodoisdisplaytheresult,usinganechostatement.However,
thisdoesn’tworkinnumeric-orientedfunctions,suchasournumerictestcondition.The
bottomlineisthatyoucannotusefloating-pointvaluesfortestconditions.
Usingstringcomparisons
Test conditions also allow you to perform comparisons on string values. Performing
comparisons on strings can get tricky, as you’ll see. Table 12.2 shows the comparison
functionsyoucanusetoevaluatetwostringvalues.
Table12.2ThetestStringComparisons
Comparison
str1=str2
str1!=str2
str1<str2
str1>str2
-nstr1
-zstr1
Description
Checksifstr1isthesameasstringstr2
Checksifstr1isnotthesameasstr2
Checksifstr1islessthanstr2
Checksifstr1isgreaterthanstr2
Checksifstr1hasalengthgreaterthanzero
Checksifstr1hasalengthofzero
Thefollowingsectionsdescribethedifferentstringcomparisonsavailable.
Lookingatstringequality
Theequalandnotequalconditionsarefairlyself-explanatorywithstrings.It’sprettyeasy
toknowwhentwostringvaluesarethesameornot:
$cattest7.sh
#!/bin/bash
#testingstringequality
testuser=rich
#
if[$USER=$testuser]
then
echo“Welcome$testuser”
fi
$
$./test7.sh
Welcomerich
$
Also,usingthenotequalsstringcomparisonallowsyoutodetermineiftwostringshave
thesamevalueornot:
$cattest8.sh
#!/bin/bash
#testingstringequality
testuser=baduser
#
if[$USER!=$testuser]
then
echo“Thisisnot$testuser”
else
echo“Welcome$testuser”
fi
$
$./test8.sh
Thisisnotbaduser
$
Keepinmindthatthetestcomparisontakesallpunctuationandcapitalizationintoaccount
whencomparingstringsforequality.
Lookingatstringorder
Tryingtodetermineifonestringislessthanorgreaterthananotheriswherethingsstart
getting tricky. Two problems often plague shell programmers when trying to use the
greater-thanorless-thanfeaturesoftestconditions:
Thegreater-thanandless-thansymbolsmustbeescaped,ortheshellusesthemas
redirectionsymbols,withthestringvaluesasfilenames.
Thegreater-thanandless-thanorderisnotthesameasthatusedwiththesort
command.
Thefirstitemcanresultinahugeproblemthatoftengoesundetectedwhenprogramming
your scripts. Here’s an example of what sometimes happens to novice shell script
programmers:
$catbadtest.sh
#!/bin/bash
#mis-usingstringcomparisons
#
val1=baseball
val2=hockey
#
if[$val1>$val2]
then
echo“$val1isgreaterthan$val2”
else
echo“$val1islessthan$val2”
fi
$
$./badtest.sh
baseballisgreaterthanhockey
$ls-lhockey
-rw-r—r—1richrich0Sep3019:08hockey
$
Byjustusingthegreater-thansymbolitselfinthescript,noerrorsaregenerated,butthe
resultsarewrong.Thescriptinterpretedthegreater-thansymbolasanoutputredirection
(seeChapter15).Thus,itcreatedafilecalledhockey.Becausetheredirectioncompleted
successfully, the test condition returns a zero exit status code, which the if statement
evaluatesasthoughthingscompletedsuccessfully!
Tofixthisproblem,youneedtoproperlyescapethegreater-thansymbol:
$cattest9.sh
#!/bin/bash
#mis-usingstringcomparisons
#
val1=baseball
val2=hockey
#
if[$val1\>$val2]
then
echo“$val1isgreaterthan$val2”
else
echo“$val1islessthan$val2”
fi
$
$./test9.sh
baseballislessthanhockey
$
Now that answer is more along the lines of what you would expect from the string
comparison.
Thesecondissueisalittlemoresubtle,andyoumaynotevenrunacrossitunlessyouare
working with uppercase and lowercase letters. The sort command handles uppercase
lettersoppositetothewaythetestconditionsconsiderthem:
$cattest9b.sh
#!/bin/bash
#testingstringsortorder
val1=Testing
val2=testing
#
if[$val1\>$val2]
then
echo“$val1isgreaterthan$val2”
else
echo“$val1islessthan$val2”
fi
$
$./test9b.sh
Testingislessthantesting
$
$sorttestfile
testing
Testing
$
Capitalizedlettersaretreatedaslessthanlowercaselettersintestcomparisons.However,
thesortcommanddoestheopposite.Whenyouputthesamestringsinafileandusethe
sort command, the lowercase letters appear first. This is due to different ordering
techniques.
Test comparisons use standard ASCII ordering, using each character’s ASCII numeric
valuetodeterminethesortorder.Thesortcommandusesthesortingorderdefinedforthe
systemlocalelanguagesettings.FortheEnglishlanguage,thelocalesettingsspecifythat
lowercaselettersappearbeforeuppercaselettersinsortedorder.
Note
Thetestcommandandtestexpressionsusethestandardmathematicalcomparison
symbolsforstringcomparisonsandtextcodesfornumericalcomparisons.Thisisa
subtlefeaturethatmanyprogrammersmanagetogetreversed.Ifyouusethe
mathematicalcomparisonsymbolsfornumericvalues,theshellinterpretsthemas
stringvaluesandmaynotproducethecorrectresults.
Lookingatstringsize
The-nand-zcomparisonsarehandywhentryingtoevaluatewhetheravariablecontains
data:
$cattest10.sh
#!/bin/bash
#testingstringlength
val1=testing
val2=”
#
if[-n$val1]
then
echo“Thestring‘$val1’isnotempty”
else
echo“Thestring‘$val1’isempty”
fi
#
if[-z$val2]
then
echo“Thestring‘$val2’isempty”
else
echo“Thestring‘$val2’isnotempty”
fi
#
if[-z$val3]
then
echo“Thestring‘$val3’isempty”
else
echo“Thestring‘$val3’isnotempty”
fi
$
$./test10.sh
Thestring‘testing’isnotempty
Thestring”isempty
Thestring”isempty
$
This example creates two string variables. The val1 variable contains a string, and the
val2variableiscreatedasanemptystring.Thefollowingcomparisonsaremadeasshown
below:
if[-n$val1]
Theprecedingcodedetermineswhethertheval1variableisnon-zeroinlength,whichitis,
soitsthensectionisprocessed.
if[-z$var2]
Thisprecedingcodedetermineswhethertheval2variableiszeroinlength,whichitis,so
itsthensectionisprocessed.
if[-z$val3]
Theprecedingdetermineswhethertheval3 variable is zero in length. This variable was
never defined in the shell script, so it indicates that the string length is still zero, even
thoughitwasn’tdefined.
Tip
Emptyanduninitializedvariablescanhavecatastrophiceffectsonyourshellscript
tests.Ifyou’renotsureofthecontentsofavariable,it’salwaysbesttotestifthe
variablecontainsavalueusing-nor-zbeforeusingitinanumericorstring
comparison.
Usingfilecomparisons
Thelastcategoryoftestcomparisonsisquitepossiblythemostpowerfulandmostused
comparisons in shell scripting. This category allows you to test the status of files and
directoriesontheLinuxfilesystem.Table12.3liststhesecomparisons.
Table12.3ThetestFileComparisons
Comparison
-dfile
-efile
-ffile
-rfile
-sfile
-wfile
-xfile
-Ofile
-Gfile
Description
Checksiffileexistsandisadirectory
Checksiffileexists
Checksiffileexistsandisafile
Checksiffileexistsandisreadable
Checksiffileexistsandisnotempty
Checksiffileexistsandiswritable
Checksiffileexistsandisexecutable
Checksiffileexistsandisownedbythecurrentuser
Checksiffileexistsandthedefaultgroupisthesameasthecurrent
user
file1-nt
file2
Checksiffile1isnewerthanfile2
file1-ot
file2
Checksiffile1isolderthanfile2
Theseconditionsgiveyoutheabilitytocheckfilesystemfileswithinshellscripts.They
areoftenusedinscriptsthataccessfiles.Becausethey’reusedsooften,let’slookateach
oftheseindividually.
Checkingdirectories
The -d test checks to see if a specified directory exists on the system. This is usually a
goodthingtodoifyou’retryingtowriteafiletoadirectoryorbeforeyoutrytochangeto
adirectorylocation:
$cattest11.sh
#!/bin/bash
#Lookbeforeyouleap
#
jump_directory=/home/arthur
#
if[-d$jump_directory]
then
echo“The$jump_directorydirectoryexists”
cd$jump_directory
ls
else
echo“The$jump_directorydirectorydoesnotexist”
fi
#
$
$./test11.sh
The/home/arthurdirectorydoesnotexist
$
The-dtestconditioncheckstoseeifthe jump_directoryvariable’sdirectoryexists.Ifit
does,itproceedstousethecdcommandtochangetothecurrentdirectoryandperformsa
directorylisting.Ifitdoesnot,thescriptemitsawarningmessageandexitsthescript.
Checkingwhetheranobjectexists
The-ecomparisonallowsyoutocheckifeitherafileordirectoryobjectexistsbeforeyou
attempttouseitinyourscript:
$cattest12.sh
#!/bin/bash
#Checkifeitheradirectoryorfileexists
#
location=$HOME
file_name=“sentinel”
#
if[-e$location]
then#Directorydoesexist
echo“OKonthe$locationdirectory.”
echo“Nowcheckingonthefile,$file_name.”
#
if[-e$location/$file_name]
then#Filedoesexist
echo“OKonthefilename”
echo“UpdatingCurrentDate…”
date>>$location/$file_name
#
else#Filedoesnotexist
echo“Filedoesnotexist”
echo“Nothingtoupdate”
fi
#
else#Directorydoesnotexist
echo“The$locationdirectorydoesnotexist.”
echo“Nothingtoupdate”
fi
#
$
$./test12.sh
OKonthe/home/Christinedirectory.
Nowcheckingonthefile,sentinel.
Filedoesnotexist
Nothingtoupdate
$
$touchsentinel
$
$./test12.sh
OKonthe/home/Christinedirectory.
Nowcheckingonthefile,sentinel.
OKonthefilename
UpdatingCurrentDate…
$
The first check uses the -e comparison to determine whether the user has a $HOME
directory. If so, the next -e comparison checks to determine whether the sentinel file
existsinthe $HOMEdirectory.Ifthefiledoesn’texist,theshellscriptnotesthatthefileis
missingandthatthereisnothingtoupdate.
Toensurethattheupdatewillwork,thesentinelfilewascreatedandtheshellscriptwas
run a second time. This time when the conditions are tested, both the $HOME and the
sentinelfilearefound,andthecurrentdateandtimeisappendedtothefile.
Checkingforafile
The -e comparison works for both files and directories. To be sure that the object
specifiedisafileandnotadirectory,youmustusethe-fcomparison:
$cattest13.sh
#!/bin/bash
#Checkifeitheradirectoryorfileexists
#
item_name=$HOME
echo
echo“Theitembeingchecked:$item_name”
echo
#
if[-e$item_name]
then#Itemdoesexist
echo“Theitem,$item_name,doesexist.”
echo“Butisitafile?”
echo
#
if[-f$item_name]
then#Itemisafile
echo“Yes,$item_nameisafile.”
#
else#Itemisnotafile
echo“No,$item_nameisnotafile.”
fi
#
else#Itemdoesnotexist
echo“Theitem,$item_name,doesnotexist.”
echo“Nothingtoupdate”
fi
#
$./test13.sh
Theitembeingchecked:/home/Christine
Theitem,/home/Christine,doesexist.
Butisitafile?
No,/home/Christineisnotafile.
$
This little script does lots of checking! First, it uses the -e comparison to test whether
$HOMEexists.Ifitdoes,ituses-ftotestwhetherit’safile.Ifitisn’tafile(whichofcourse
itisn’t),amessageisdisplayedstatingthatitisnotafile.
Aslightmodificationtothevariable,item_name,replacingthedirectory$HOMEwithafile,
$HOME/sentinel,causesadifferentoutcome:
$nanotest13.sh
$
$cattest13.sh
#!/bin/bash
#Checkifeitheradirectoryorfileexists
#
item_name=$HOME/sentinel
[…]
$
$./test13.sh
Theitembeingchecked:/home/Christine/sentinel
Theitem,/home/Christine/sentinel,doesexist.
Butisitafile?
Yes,/home/Christine/sentinelisafile.
$
The test13.shscriptlistingissnipped,becausetheonlyitemchangedintheshellscript
was the item_name variable’s value. Now when the script is run, the -f test on
$HOME/sentinel exits with a zero status, triggering the then statement, which in turn
outputsthemessageYes,/home/Christine/sentinelisafile.
Checkingforreadaccess
Beforetryingtoreaddatafromafile,it’susuallyagoodideatotestwhetheryoucanread
fromthefilefirst.Youdothiswiththe-rcomparison:
$cattest14.sh
#!/bin/bash
#testingifyoucanreadafile
pwfile=/etc/shadow
#
#first,testifthefileexists,andisafile
if[-f$pwfile]
then
#nowtestifyoucanreadit
if[-r$pwfile]
then
tail$pwfile
else
echo“Sorry,Iamunabletoreadthe$pwfilefile”
fi
else
echo“Sorry,thefile$filedoesnotexist”
fi
$
$./test14.sh
Sorry,Iamunabletoreadthe/etc/shadowfile
$
The /etc/shadow file contains the encrypted passwords for system users, so it’s not
readablebynormalusersonthesystem.The-rcomparisondeterminedthatreadaccessto
thefilewasn’tallowed,sothetestcommandfailedandthebashshellexecutedthe else
sectionoftheif-thenstatement.
Checkingforemptyfiles
Youshoulduse -s comparison to check whether a file is empty, especially if you don’t
wanttoremoveanon-emptyfile.Becarefulbecausewhenthe-scomparisonsucceeds,it
indicatesthatafilehasdatainit:
$cattest15.sh
#!/bin/bash
#Testingifafileisempty
#
file_name=$HOME/sentinel
#
if[-f$file_name]
then
if[-s$file_name]
then
echo”The$file_namefileexistsandhasdatainit.”
echo”Willnotremovethisfile.”
#
else
echo”The$file_namefileexists,butisempty.”
echo”Deletingemptyfile…”
rm$file_name
fi
else
echo”File,$file_name,doesnotexist.”
fi
#
$ls-l$HOME/sentinel
-rw-rw-r—.1ChristineChristine29Jun2505:32/home/Christine/sentinel
$
$./test15.sh
The/home/Christine/sentinelfileexistsandhasdatainit.
Willnotremovethisfile.
$
First,the-fcomparisontestswhetherthefileexists.Ifitdoesexist,the -scomparisonis
triggeredtodeterminewhetherthefileisempty.Anemptyfilewillbedeleted.Youcan
seefromthe ls-lthatthe sentinelfileisnotempty,andthereforethescriptdoesnot
deleteit.
Checkingwhetheryoucanwritetoafile
The -w comparison determines whether you have permission to write to a file. The
test16.sh script is simply an update of the test13.sh script. Now instead of just
checkingwhetherthe item_nameexistsandisafile,thescriptalsocheckstoseewhether
ithaspermissiontowritetothefile:
$cattest16.sh
#!/bin/bash
#Checkifafileiswritable.
#
item_name=$HOME/sentinel
echo
echo“Theitembeingchecked:$item_name”
echo
[…]
echo“Yes,$item_nameisafile.”
echo“Butisitwritable?”
echo
#
if[-w$item_name]
then#Itemiswritable
echo“Writingcurrenttimeto$item_name”
date+%H%M>>$item_name
#
else#Itemisnotwritable
echo“Unabletowriteto$item_name”
fi
#
else#Itemisnotafile
echo“No,$item_nameisnotafile.”
fi
[…]
$
$ls-lsentinel
-rw-rw-r—.1ChristineChristine0Jun2705:38sentinel
$
$./test16.sh
Theitembeingchecked:/home/Christine/sentinel
Theitem,/home/Christine/sentinel,doesexist.
Butisitafile?
Yes,/home/Christine/sentinelisafile.
Butisitwritable?
Writingcurrenttimeto/home/Christine/sentinel
$
$catsentinel
0543
$
The item_namevariableissetto $HOME/sentinel, and this file allows user write access
(seeChapter7formoreinformationonfilepermissions).Thus,whenthescriptisrun,the
-w test expressions returns a non-zero exit status and the then code block is executed,
whichwritesatimestampintothesentinelfile.
When the sentinel file user’s write access is removed via chmod, the -w test expression
returnsanon-zerostatus,andatimestampisnotwrittentothefile:
$chmodu-wsentinel
$
$ls-lsentinel
-r—rw-r—.1ChristineChristine5Jun2705:43sentinel
$
$./test16.sh
Theitembeingchecked:/home/Christine/sentinel
Theitem,/home/Christine/sentinel,doesexist.
Butisitafile?
Yes,/home/Christine/sentinelisafile.
Butisitwritable?
Unabletowriteto/home/Christine/sentinel
$
The chmodcommandcouldbeusedagaintograntthewritepermissionbackfortheuser.
This would make the write test expression return a zero exit status and allow a write
attempttothefile.
Checkingwhetheryoucanrunafile
The-xcomparisonisahandywaytodeterminewhetheryouhaveexecutepermissionfor
a specific file. Although this may not be needed for most commands, if you run lots of
scriptsfromyourshellscripts,itcouldbeuseful:
$cattest17.sh
#!/bin/bash
#testingfileexecution
#
if[-xtest16.sh]
then
echo“Youcanrunthescript:“
./test16.sh
else
echo“Sorry,youareunabletoexecutethescript”
fi
$
$./test17.sh
Youcanrunthescript:
[…]
$
$chmodu-xtest16.sh
$
$./test17.sh
Sorry,youareunabletoexecutethescript
$
Thisexampleshellscriptusesthe -xcomparisontotestwhetheryouhavepermissionto
execute the test16.sh script. If so, it runs the script. After successfully running the
test16.sh script the first time, the permissions were changed. This time, the -x
comparison failed, because execute permission had been removed for the test16.sh
script.
Checkingownership
The-Ocomparisonallowsyoutoeasilytestwhetheryou’retheownerofafile:
$cattest18.sh
#!/bin/bash
#checkfileownership
#
if[-O/etc/passwd]
then
echo“Youaretheownerofthe/etc/passwdfile”
else
echo“Sorry,youarenottheownerofthe/etc/passwdfile”
fi
$
$./test18.sh
Sorry,youarenottheownerofthe/etc/passwdfile
$
Thescriptusesthe -Ocomparisontotestwhethertheuserrunningthescriptistheowner
ofthe/etc/passwdfile.Thescriptisrununderanormaluseraccount,sothetestfails.
Checkingdefaultgroupmembership
The -G comparison checks the default group of a file, and it succeeds if it matches the
groupofthedefaultgroupfortheuser.Thiscanbesomewhatconfusingbecausethe -G
comparison checks the default groups only and not all the groups to which the user
belongs.Here’sanexampleofthis:
$cattest19.sh
#!/bin/bash
#checkfilegrouptest
#
if[-G$HOME/testing]
then
echo“Youareinthesamegroupasthefile”
else
echo“Thefileisnotownedbyyourgroup”
fi
$
$ls-l$HOME/testing
-rw-rw-r—1richrich582014-07-3015:51/home/rich/testing
$
$./test19.sh
Youareinthesamegroupasthefile
$
$chgrpsharing$HOME/testing
$
$./test19
Thefileisnotownedbyyourgroup
$
Thefirsttimethescriptisrun,the $HOME/testingfileisinthe richgroup,andthe -G
comparisonsucceeds.Next,thegroupischangedtothe sharinggroup,ofwhichtheuser
isalsoamember.However,the-Gcomparisonfailed,becauseitcomparesonlythedefault
groups,notanyadditionalgroupmemberships.
Checkingfiledate
The last set of comparisons deal with comparing the creation times of two files. This
comes in handy when writing scripts to install software. Sometimes, you don’t want to
installafilethatisolderthanafilealreadyinstalledonthesystem.
The-ntcomparisondetermineswhetherafileisnewerthananotherfile.Ifafileisnewer,
ithasamorerecentfilecreationtime.The -otcomparisondetermineswhetherafileis
olderthananotherfile.Ifthefileisolder,ithasanolderfilecreationtime:
$cattest20.sh
#!/bin/bash
#testingfiledates
#
if[test19.sh-nttest18.sh]
then
echo“Thetest19fileisnewerthantest18”
else
echo“Thetest18fileisnewerthantest19”
fi
if[test17.sh-ottest19.sh]
then
echo“Thetest17fileisolderthanthetest19file”
fi
$
$./test20.sh
Thetest19fileisnewerthantest18
Thetest17fileisolderthanthetest19file
$
$ls-ltest17.shtest18.shtest19.sh
-rwxrw-r—1richrich1672014-07-3016:31test17.sh
-rwxrw-r—1richrich1852014-07-3017:46test18.sh
-rwxrw-r—1richrich1672014-07-3017:50test19.sh
$
Thefilepathsusedinthecomparisonsarerelativetothedirectoryfromwhichyourunthe
script. This can cause problems if the files being checked are moved around. Another
problemisthatneitherofthesecomparisonscheckswhetherthefileexistsfirst.Trythis
test:
$cattest21.sh
#!/bin/bash
#testingfiledates
#
if[badfile1-ntbadfile2]
then
echo“Thebadfile1fileisnewerthanbadfile2”
else
echo“Thebadfile2fileisnewerthanbadfile1”
fi
$
$./test21.sh
Thebadfile2fileisnewerthanbadfile1
$
This little example demonstrates that if the files don’t exist, the -nt comparison just
returnsafailedcondition.It’simperativetoensurethatthefilesexistbeforetryingtouse
theminthe-ntor-otcomparison.
ConsideringCompoundTesting
The if-then statement allows you to use Boolean logic to combine tests. You can use
thesetwoBooleanoperators:
[condition1]&&[condition2]
[condition1]||[condition2]
The first Boolean operation uses the AND Boolean operator to combine two conditions.
Bothconditionsmustbemetforthethensectiontoexecute.
Tip
BooleanlogicisamethodthatreducesthepotentialreturnedvaluestobeeitherTRUE
orFALSE.
ThesecondBooleanoperationusestheORBooleanoperatortocombinetwoconditions.If
eitherconditionevaluatestoaTRUEcondition,thethensectionisexecuted.
ThefollowingshowstheANDBooleanoperatorinuse:
$cattest22.sh
#!/bin/bash
#testingcompoundcomparisons
#
if[-d$HOME]&&[-w$HOME/testing]
then
echo“Thefileexistsandyoucanwritetoit”
else
echo“Icannotwritetothefile”
fi
$
$./test22.sh
Icannotwritetothefile
$
$touch$HOME/testing
$
$./test22.sh
Thefileexistsandyoucanwritetoit
$
Using the AND Boolean operator, both of the comparisons must be met. The first
comparison checks to see if the $HOME directory exists for the user. The second
comparisoncheckstoseeifthere’safilecalled testingintheuser’s $HOMEdirectory,
andiftheuserhaswritepermissionsforthefile.Ifeitherofthesecomparisonsfails,the
if statement fails and the shell executes the else section. If both of the comparisons
succeed,theifstatementsucceeds,andtheshellexecutesthethensection.
WorkingwithAdvancedif-thenFeatures
Two additions to the bash shell provide advanced features that you can use in if-then
statements:
Doubleparenthesesformathematicalexpressions
Doublesquarebracketsforadvancedstringhandlingfunctions
Thefollowingsectionsdescribeeachofthesefeaturesinmoredetail.
Usingdoubleparentheses
The double parentheses command allows you to incorporate advanced mathematical
formulas in your comparisons. The test command allows for only simple arithmetic
operations in the comparison. The double parentheses command provides more
mathematicalsymbols,whichprogrammerswhohaveusedotherprogramminglanguages
maybefamiliarwithusing.Here’stheformatofthedoubleparenthesescommand:
((expression))
The expression term can be any mathematical assignment or comparison expression.
Besides the standard mathematical operators that the test command uses, Table 12.4
shows the list of additional operators available for use in the double parentheses
command.
Table12.4TheDoubleParenthesesCommandSymbols
Symbol
val++
val—
++val
—val
!
∼
**
<<
>>
&
|
&&
||
Description
Post-increment
Post-decrement
Pre-increment
Pre-decrement
Logicalnegation
Bitwisenegation
Exponentiation
Leftbitwiseshift
Rightbitwiseshift
BitwiseBooleanAND
BitwiseBooleanOR
LogicalAND
LogicalOR
Youcanusethedoubleparenthesescommandinan if statement,aswellasinanormal
commandinthescriptforassigningvalues:
$cattest23.sh
#!/bin/bash
#usingdoubleparenthesis
#
val1=10
#
if(($val1**2>90))
then
((val2=$val1**2))
echo“Thesquareof$val1is$val2”
fi
$
$./test23.sh
Thesquareof10is100
$
Noticethatyoudon’tneedtoescapethegreater-thansymbolintheexpressionwithinthe
double parentheses. This is yet another advanced feature besides the double parentheses
command.
Usingdoublebrackets
Thedoublebracketcommandprovidesadvancedfeaturesforstringcomparisons.Here’s
thedoublebracketcommandformat:
[[expression]]
The double bracketed expression uses the standard string comparison used in the test
evaluations.However,itprovidesanadditionalfeaturethatthetestevaluationsdon’t—
patternmatching.
Note
Doublebracketsworkfineinthebashshell.Beaware,however,thatnotallshells
supportdoublebrackets.
In pattern matching, you can define a regular expression (discussed in detail in Chapter
20)that’smatchedagainstthestringvalue:
$cattest24.sh
#!/bin/bash
#usingpatternmatching
#
if[[$USER==r*]]
then
echo“Hello$USER”
else
echo“Sorry,Idonotknowyou”
fi
$
$./test24.sh
Hellorich
$
Noticeintheprecedingscriptthatdoubleequalsigns(==)areused.Thesedoubleequal
signs designate the string to the right (r*) as a pattern, and pattern matching rules are
applied. The double bracket command matches the $USER environment variable to see
whetheritstartswiththeletter r.Ifso,thecomparisonsucceeds,andtheshellexecutes
thethensectioncommands.
ConsideringthecaseCommand
Often, you’ll find yourself trying to evaluate a variable’s value, looking for a specific
valuewithinasetofpossiblevalues.Inthisscenario,youenduphavingtowritealengthy
if-then-elsestatement,likethis:
$cattest25.sh
#!/bin/bash
#lookingforapossiblevalue
#
if[$USER=“rich”]
then
echo“Welcome$USER”
echo“Pleaseenjoyyourvisit”
elif[$USER=“barbara”]
then
echo“Welcome$USER”
echo“Pleaseenjoyyourvisit”
elif[$USER=“testing”]
then
echo“Specialtestingaccount”
elif[$USER=“jessica”]
then
echo“Donotforgettologoutwhenyou’redone”
else
echo“Sorry,youarenotallowedhere”
fi
$
$./test25.sh
Welcomerich
Pleaseenjoyyourvisit
$
The elifstatementscontinuethe if-thenchecking,lookingforaspecificvalueforthe
singlecomparisonvariable.
Insteadofhavingtowriteallthe elifstatementstocontinuecheckingthesamevariable
value, you can use the casecommand.The case command checks multiple values of a
singlevariableinalist-orientedformat:
casevariablein
pattern1|pattern2)commands1;;
pattern3)commands2;;
*)defaultcommands;;
esac
The case command compares the variable specified against the different patterns. If the
variable matches the pattern, the shell executes the commands specified for the pattern.
You can list more than one pattern on a line, using the bar operator to separate each
pattern.Theasterisksymbolisthecatch-allforvaluesthatdon’tmatchanyofthelisted
patterns. Here’s an example of converting the if-then-else program to using the case
command:
$cattest26.sh
#!/bin/bash
#usingthecasecommand
#
case$USERin
rich|barbara)
echo“Welcome,$USER”
echo“Pleaseenjoyyourvisit”;;
testing)
echo“Specialtestingaccount”;;
jessica)
echo“Donotforgettologoffwhenyou’redone”;;
*)
echo“Sorry,youarenotallowedhere”;;
esac
$
$./test26.sh
Welcome,rich
Pleaseenjoyyourvisit
$
The case command provides a much cleaner way of specifying the various options for
eachpossiblevariablevalue.
Summary
Structured commands allow you to alter the normal flow of shell script execution. The
most basic structured command is the if-then statement. This statement provides a
command evaluation and performs other commands based on the evaluated command’s
output.
You can expand the if-then statement to include a set of commands the bash shell
executes if the specified command fails as well. The if-then-else statement executes
commandsonlyifthecommandbeingevaluatedreturnsanon-zeroexitstatuscode.
Youcanalsolink if-then-elsestatementstogether,usingthe elifstatement.The elif
isequivalenttousinganelseifstatement,providingforadditionalcheckingofwhether
theoriginalcommandthatwasevaluatedfailed.
Inmostscripts,insteadofevaluatingacommand,you’llwanttoevaluateacondition,such
asanumericvalue,thecontentsofastring,orthestatusofafileordirectory.The test
commandprovidesaneasywayforyoutoevaluatealltheseconditions.Ifthecondition
evaluatestoa TRUEcondition,the testcommandproducesazeroexitstatuscodeforthe
if-then statement. If the condition evaluates to a FALSE condition, the test command
producesanon-zeroexitstatuscodefortheif-thenstatement.
Thesquarebracketisaspecialbashcommandthatisasynonymforthe testcommand.
You can enclose a test condition in square brackets in the if-then statement to test for
numeric,string,andfileconditions.
The double parentheses command provides advanced mathematical evaluations using
additional operators. The double square bracket command allows you to perform
advancedstringpattern-matchingevaluations.
Finally,thechapterdiscussedthecasecommand,whichisashorthandwayofperforming
multipleif-then-elsecommands,checkingthevalueofasinglevariableagainstalistof
values.
Thenextchaptercontinuesthediscussionofstructuredcommandsbyexaminingtheshell
loopingcommands.Theforandwhilecommandsletyoucreateloopsthatiteratethrough
commandsforagivenperiodoftime.
Chapter13
MoreStructuredCommands
InThisChapter
1. Loopingwiththeforstatement
2. Iteratingwiththeuntilstatement
3. Usingthewhilestatement
4. Combiningloops
5. Redirectingloopoutput
Inthepreviouschapter,yousawhowtomanipulatetheflowofashellscriptprogram
bycheckingtheoutputofcommandsandthevaluesofvariables.Inthischapter,we
continuetolookatstructuredcommandsthatcontroltheflowofyourshellscripts.
You’llseehowyoucanperformrepeatingprocesses,commandsthatcanloop
throughasetofcommandsuntilanindicatedconditionhasbeenmet.Thischapter
discussesanddemonstratesthefor,while,anduntilbashshellloopingcommands.
TheforCommand
Iterating through a series of commands is a common programming practice. Often, you
need to repeat a set of commands until a specific condition has been met, such as
processingallthefilesinadirectory,alltheusersonasystem,orallthelinesinatextfile.
The bash shell provides the for command to allow you to create a loop that iterates
throughaseriesofvalues.Eachiterationperformsadefinedsetofcommandsusingoneof
thevaluesintheseries.Here’sthebasicformatofthebashshellforcommand:
forvarinlist
do
commands
done
Yousupplytheseriesofvaluesusedintheiterationsinthelistparameter.Youcanspecify
thevaluesinthelistinseveralways.
Ineachiteration,thevariablevarcontainsthecurrentvalueinthelist.Thefirstiteration
usesthefirstiteminthelist,theseconditerationtheseconditem,andsoonuntilallthe
itemsinthelisthavebeenused.
Thecommandsenteredbetweenthe doand donestatementscanbeoneormorestandard
bash shell commands. Within the commands, the $var variable contains the current list
itemvaluefortheiteration.
Note
Ifyouprefer,youcanincludethedostatementonthesamelineastheforstatement,
butyoumustseparateitfromthelistitemsusingasemicolon:forvarinlist;
do.
We mentioned that there are several different ways to specify the values in the list. The
followingsectionsshowthevariouswaystodothat.
Readingvaluesinalist
Themostbasicuseoftheforcommandistoiteratethroughalistofvaluesdefinedwithin
theforcommanditself:
$cattest1
#!/bin/bash
#basicforcommand
fortestinAlabamaAlaskaArizonaArkansasCaliforniaColorado
do
echoThenextstateis$test
done
$./test1
ThenextstateisAlabama
ThenextstateisAlaska
ThenextstateisArizona
ThenextstateisArkansas
ThenextstateisCalifornia
ThenextstateisColorado
$
Each time the for command iterates through the list of values provided, it assigns the
$testvariablethenextvalueinthelist.The$testvariablecanbeusedjustlikeanyother
script variable within the for command statements. After the last iteration, the $test
variable remains valid throughout the remainder of the shell script. It retains the last
iterationvalue(unlessyouchangeitsvalue):
$cattest1b
#!/bin/bash
#testingtheforvariableafterthelooping
fortestinAlabamaAlaskaArizonaArkansasCaliforniaColorado
do
echo“Thenextstateis$test”
done
echo“Thelaststatewevisitedwas$test”
test=Connecticut
echo“Wait,nowwe’revisiting$test”
$./test1b
ThenextstateisAlabama
ThenextstateisAlaska
ThenextstateisArizona
ThenextstateisArkansas
ThenextstateisCalifornia
ThenextstateisColorado
ThelaststatewevisitedwasColorado
Wait,nowwe’revisitingConnecticut
$
The$testvariableretaineditsvalueandallowedustochangethevalueanduseitoutside
oftheforcommandloop,asanyothervariablewould.
Readingcomplexvaluesinalist
Thingsaren’talwaysaseasyastheyseemwiththe forloop.Therearetimeswhenyou
runintodatathatcausesproblems.Here’saclassicexampleofwhatcancauseproblems
forshellscriptprogrammers:
$catbadtest1
#!/bin/bash
#anotherexampleofhownottousetheforcommand
fortestinIdon’tknowifthis’llwork
do
echo“word:$test”
done
$./badtest1
word:I
word:dontknowifthisll
word:work
$
Ouch, that hurts. The shell saw the single quotation marks within the list values and
attemptedtousethemtodefineasingledatavalue,anditreallymessedthingsupinthe
process.
Youhavetwowaystosolvethisproblem:
Usetheescapecharacter(thebackslash)toescapethesinglequotationmark.
Usedoublequotationmarkstodefinethevaluesthatusesinglequotationmarks.
Neithersolutionisallthatfantastic,buteachonehelpssolvetheproblem:
$cattest2
#!/bin/bash
#anotherexampleofhownottousetheforcommand
fortestinIdon'tknowif“this’ll”work
do
echo“word:$test”
done
$./test2
word:I
word:don’t
word:know
word:if
word:this’ll
word:work
$
In the first problem value, you added the backslash character to escape the single
quotationmarkinthedon’tvalue.Inthesecondproblemvalue,youenclosedthethis’ll
valueindoublequotationmarks.Bothmethodsworkedfinetodistinguishthevalue.
Another problem you may run into is multi-word values. Remember that the for loop
assumes that each value is separated with a space. If you have data values that contain
spaces,yourunintoyetanotherproblem:
$catbadtest2
#!/bin/bash
#anotherexampleofhownottousetheforcommand
fortestinNevadaNewHampshireNewMexicoNewYorkNorthCarolina
do
echo“Nowgoingto$test”
done
$./badtest1
NowgoingtoNevada
NowgoingtoNew
NowgoingtoHampshire
NowgoingtoNew
NowgoingtoMexico
NowgoingtoNew
NowgoingtoYork
NowgoingtoNorth
NowgoingtoCarolina
$
Oops,that’snotexactlywhatwewanted.The forcommandseparateseachvalueinthe
listwithaspace.Iftherearespacesintheindividualdatavalues,youmustaccommodate
themusingdoublequotationmarks:
$cattest3
#!/bin/bash
#anexampleofhowtoproperlydefinevalues
fortestinNevada“NewHampshire”“NewMexico”“NewYork”
do
echo“Nowgoingto$test”
done
$./test3
NowgoingtoNevada
NowgoingtoNewHampshire
NowgoingtoNewMexico
NowgoingtoNewYork
$
Nowtheforcommandcanproperlydistinguishbetweenthedifferentvalues.Also,notice
that when you use double quotation marks around a value, the shell doesn’t include the
quotationmarksaspartofthevalue.
Readingalistfromavariable
Often what happens in a shell script is that you accumulate a list of values stored in a
variableandthenneedtoiteratethroughthelist.Youcandothisusingthe forcommand
aswell:
$cattest4
#!/bin/bash
#usingavariabletoholdthelist
list=“AlabamaAlaskaArizonaArkansasColorado”
list=$list”Connecticut”
forstatein$list
do
echo“Haveyouevervisited$state?”
done
$./test4
HaveyouevervisitedAlabama?
HaveyouevervisitedAlaska?
HaveyouevervisitedArizona?
HaveyouevervisitedArkansas?
HaveyouevervisitedColorado?
HaveyouevervisitedConnecticut?
$
The$listvariablecontainsthestandardtextlistofvaluestousefortheiterations.Notice
thatthecodealsousesanotherassignmentstatementtoadd(orconcatenate)anitemtothe
existinglistcontainedinthe $listvariable.Thisisacommonmethodforaddingtextto
theendofanexistingtextstringstoredinavariable.
Readingvaluesfromacommand
Anotherwaytogeneratevaluesforuseinthelististousetheoutputofacommand.You
usecommandsubstitutiontoexecuteanycommandthatproducesoutputandthenusethe
outputofthecommandintheforcommand:
$cattest5
#!/bin/bash
#readingvaluesfromafile
file=“states”
forstatein$(cat$file)
do
echo“Visitbeautiful$state”
done
$catstates
Alabama
Alaska
Arizona
Arkansas
Colorado
Connecticut
Delaware
Florida
Georgia
$./test5
VisitbeautifulAlabama
VisitbeautifulAlaska
VisitbeautifulArizona
VisitbeautifulArkansas
VisitbeautifulColorado
VisitbeautifulConnecticut
VisitbeautifulDelaware
VisitbeautifulFlorida
VisitbeautifulGeorgia
$
Thisexampleusesthe catcommandinthecommandsubstitutiontodisplaythecontents
of the file states. Notice that the states file includes each state on a separate line, not
separated by spaces. The for command still iterates through the output of the cat
commandonelineatatime,assumingthateachstateisonaseparateline.However,this
doesn’tsolvetheproblemofhavingspacesindata.Ifyoulistastatewithaspaceinit,the
forcommandstilltakeseachwordasaseparatevalue.There’sareasonforthis,whichwe
lookatinthenextsection.
Note
Thetest5codeexampleassignedthefilenametothevariableusingjustthefilename
withoutapath.Thisrequiresthatthefilebeinthesamedirectoryasthescript.Ifthis
isn’tthecase,youneedtouseafullpathname(eitherabsoluteorrelative)to
referencethefilelocation.
Changingthefieldseparator
ThecauseofthisproblemisthespecialenvironmentvariableIFS,calledtheinternalfield
separator.TheIFSenvironmentvariabledefinesalistofcharactersthebashshellusesas
field separators. By default, the bash shell considers the following characters as field
separators:
Aspace
Atab
Anewline
Ifthebashshellseesanyofthesecharactersinthedata,itassumesthatyou’restartinga
new data field in the list. When working with data that can contain spaces (such as
filenames),thiscanbeannoying,asyousawinthepreviousscriptexample.
Tosolvethisproblem,youcantemporarilychangetheIFSenvironmentvariablevaluesin
yourshellscripttorestrictthecharactersthebashshellrecognizesasfieldseparators.For
example,ifyouwanttochangetheIFSvaluetorecognizeonlythenewlinecharacter,you
needtodothis:
IFS=$’\n’
Addingthisstatementtoyourscripttellsthebashshelltoignorespacesandtabsindata
values.Applyingthistechniquetothepreviousscriptyieldsthefollowing:
$cattest5b
#!/bin/bash
#readingvaluesfromafile
file=“states”
IFS=$’\n’
forstatein$(cat$file)
do
echo“Visitbeautiful$state”
done
$./test5b
VisitbeautifulAlabama
VisitbeautifulAlaska
VisitbeautifulArizona
VisitbeautifulArkansas
VisitbeautifulColorado
VisitbeautifulConnecticut
VisitbeautifulDelaware
VisitbeautifulFlorida
VisitbeautifulGeorgia
VisitbeautifulNewYork
VisitbeautifulNewHampshire
VisitbeautifulNorthCarolina
$
Nowtheshellscriptcanusevaluesinthelistthatcontainspaces.
Caution
Whenworkingonlongscripts,it’spossibletochangetheIFSvalueinoneplace,and
thenforgetaboutitandassumethedefaultvalueelsewhereinthescript.Asafe
practicetogetintoistosavetheoriginalIFSvaluebeforechangingitandthen
restoreitwhenyou’refinished.
Thistechniquecanbecodedlikethis:
IFS.OLD=$IFS
IFS=$‘∖n’
<usethenewIFSvalueincode>
IFS=$IFS.OLD
ThisensuresthattheIFSvalueisreturnedtothedefaultvalueforfutureoperations
withinthescript.
Other excellent applications of the IFS environment variable are possible. Suppose you
want to iterate through values in a file that are separated by a colon (such as in the
/etc/passwdfile).YoujustneedtosettheIFSvaluetoacolon:
IFS=:
If you want to specify more than one IFS character, just string them together on the
assignmentline:
IFS=$’\n’:;”
Thisassignmentusesthenewline,colon,semicolon,anddoublequotationmarkcharacters
as field separators. There’s no limit to how you can parse your data using the IFS
characters.
Readingadirectoryusingwildcards
Finally,youcanusetheforcommandtoautomaticallyiteratethroughadirectoryoffiles.
Todothis,youmustuseawildcardcharacterinthefileorpathname.Thisforcestheshell
tousefileglobbing.Fileglobbingistheprocessofproducingfilenamesorpathnamesthat
matchaspecifiedwildcardcharacter.
This feature is great for processing files in a directory when you don’t know all the
filenames:
$cattest6
#!/bin/bash
#iteratethroughallthefilesinadirectory
forfilein/home/rich/test/*
do
if[-d“$file”]
then
echo“$fileisadirectory”
elif[-f“$file”]
then
echo“$fileisafile”
fi
done
$./test6
/home/rich/test/dir1isadirectory
/home/rich/test/myprog.cisafile
/home/rich/test/myprogisafile
/home/rich/test/myscriptisafile
/home/rich/test/newdirisadirectory
/home/rich/test/newfileisafile
/home/rich/test/newfile2isafile
/home/rich/test/testdirisadirectory
/home/rich/test/testingisafile
/home/rich/test/testprogisafile
/home/rich/test/testprog.cisafile
$
Theforcommanditeratesthroughtheresultsofthe/home/rich/test/*listing.Thecode
testseachentryusingthe testcommand(usingthesquarebracketmethod)toseeifit’sa
directory,usingthe-dparameter,orafile,usingthe-fparameter(SeeChapter12).
Noticeinthisexamplethatwedidsomethingdifferentintheifstatementtests:
if[-d“$file”]
In Linux, it’s perfectly legal to have directory and filenames that contain spaces. To
accommodate that, you should enclose the $file variable in double quotation marks. If
youdon’t,you’llgetanerrorifyourunintoadirectoryorfilenamethatcontainsspaces:
./test6:line6:[:toomanyarguments
./test6:line9:[:toomanyarguments
The bash shell interprets the additional words as arguments within the test command,
causinganerror.
Youcanalsocombineboththedirectorysearchmethodandthelistmethodinthesame
forstatementbylistingaseriesofdirectorywildcardsintheforcommand:
$cattest7
#!/bin/bash
#iteratingthroughmultipledirectories
forfilein/home/rich/.b*/home/rich/badtest
do
if[-d“$file”]
then
echo“$fileisadirectory”
elif[-f“$file”]
then
echo“$fileisafile”
else
echo“$filedoesn’texist”
fi
done
$./test7
/home/rich/.backup.timestampisafile
/home/rich/.bash_historyisafile
/home/rich/.bash_logoutisafile
/home/rich/.bash_profileisafile
/home/rich/.bashrcisafile
/home/rich/badtestdoesn’texist
$
Theforstatementfirstusesfileglobbingtoiteratethroughthelistoffilesthatresultfrom
thewildcardcharacter;thenititeratesthroughthenextfileinthelist.Youcancombine
anynumberofwildcardentriesinthelisttoiteratethrough.
Caution
Noticethatyoucanenteranythinginthelistdata.Evenifthefileordirectorydoesn’t
exist,theforstatementattemptstoprocesswhateveryouplaceinthelist.Thiscan
beaproblemwhenworkingwithfilesanddirectories.Youhavenowayofknowing
ifyou’retryingtoiteratethroughanonexistentdirectory:It’salwaysagoodideato
testeachfileordirectorybeforetryingtoprocessit.
TheC-StyleforCommand
If you’ve done any programming using the C programming language, you’re probably
surprisedbythewaythebashshellusesthe forcommand.IntheClanguage,a forloop
normally defines a variable, which it then alters automatically during each iteration.
Typically,programmersusethisvariableasacounterandeitherincrementordecrement
the counter by one in each iteration. The bash for command can also provide this
functionality.ThissectionshowsyouhowtouseaC-style forcommandinabashshell
script.
TheClanguageforcommand
TheClanguageforcommandhasaspecificmethodforspecifyingavariable,acondition
thatmustremaintruefortheiterationstocontinue,andamethodforalteringthevariable
for each iteration. When the specified condition becomes false, the for loop stops. The
conditionequationisdefinedusingstandardmathematicalsymbols.Forexample,consider
thefollowingClanguagecode:
for(i=0;i<10;i++)
{
printf(“Thenextnumberis%d\n”,i);
}
Thiscodeproducesasimpleiterationloop,wherethevariable iisusedasacounter.The
first section assigns a default value to the variable. The middle section defines the
conditionunderwhichtheloopwilliterate.Whenthedefinedconditionbecomesfalse,the
for loop stops iterations. The last section defines the iteration process. After each
iteration, the expression defined in the last section is executed. In this example, the i
variableisincrementedbyoneaftereachiteration.
ThebashshellalsosupportsaversionoftheforloopthatlookssimilartotheC-stylefor
loop,althoughitdoeshavesomesubtledifferences,includingacoupleofthingsthatwill
confuseshellscriptprogrammers.Here’sthebasicformatoftheC-stylebashforloop:
for((variableassignment;condition;iterationprocess))
TheformatoftheC-style for loop can be confusing for bash shell script programmers,
because it uses C-style variable references instead of the shell-style variable references.
Here’swhataC-styleforcommandlookslike:
for((a=1;a<10;a++))
Notice that there are a couple of things that don’t follow the standard bash shell for
method:
Theassignmentofthevariablevaluecancontainspaces.
Thevariableintheconditionisn’tprecededwithadollarsign.
Theequationfortheiterationprocessdoesn’tusetheexprcommandformat.
The shell developers created this format to more closely resemble the C-style for
command. Although this is great for C programmers, it can throw even expert shell
programmersintoatizzy.BecarefulwhenusingtheC-styleforloopinyourscripts.
Here’sanexampleofusingtheC-styleforcommandinabashshellprogram:
$cattest8
#!/bin/bash
#testingtheC-styleforloop
for((i=1;i<=10;i++))
do
echo“Thenextnumberis$i”
done
$./test8
Thenextnumberis1
Thenextnumberis2
Thenextnumberis3
Thenextnumberis4
Thenextnumberis5
Thenextnumberis6
Thenextnumberis7
Thenextnumberis8
Thenextnumberis9
Thenextnumberis10
$
The for loop iterates through the commands using the variable defined in the for loop
(theletteriinthisexample).Ineachiteration,the $ivariablecontainsthevalueassigned
inthe forloop.Aftereachiteration,theloopiterationprocessisappliedtothevariable,
whichinthisexample,incrementsthevariablebyone.
Usingmultiplevariables
TheC-style forcommandalsoallowsyoutousemultiplevariablesfortheiteration.The
loophandleseachvariableseparately,allowingyoutodefineadifferentiterationprocess
for each variable. Although you can have multiple variables, you can define only one
conditionintheforloop:
$cattest9
#!/bin/bash
#multiplevariables
for((a=1,b=10;a<=10;a++,b—))
do
echo“$a-$b”
done
$./test9
1-10
2-9
3-8
4-7
5-6
6-5
7-4
8-3
9-2
10-1
$
The a and b variables are each initialized with different values, and different iteration
processesaredefined.Whiletheloopincreasesthe avariable,itdecreasesthe bvariable
foreachiteration.
ThewhileCommand
The whilecommandissomewhatofacrossbetweenthe if-thenstatementandthe for
loop.Thewhilecommandallowsyoutodefineacommandtotestandthenloopthrough
asetofcommandsforaslongasthedefinedtestcommandreturnsazeroexitstatus.It
teststhe testcommandatthestartofeachiteration.Whenthe testcommandreturnsa
non-zeroexitstatus,thewhilecommandstopsexecutingthesetofcommands.
Basicwhileformat
Here’sfheformatofthewhilecommand:
whiletestcommand
do
othercommands
done
Thetestcommanddefinedinthe whilecommandistheexactsameformatasin if-then
statements(seeChapter12).Asinthe if-thenstatement,youcanuseanynormal bash
shellcommand,oryoucanusethe testcommandtotestforconditions,suchasvariable
values.
Thekeytothe whilecommandisthattheexitstatusofthe testcommandspecifiedmust
change,basedonthecommandsrunduringtheloop.Iftheexitstatusneverchanges,the
whileloopwillgetstuckinaninfiniteloop.
Themostcommonuseofthe testcommandistousebracketstocheckavalueofashell
variablethat’susedintheloopcommands:
$cattest10
#!/bin/bash
#whilecommandtest
var1=10
while[$var1-gt0]
do
echo$var1
var1=$[$var1-1]
done
$./test10
10
9
8
7
6
5
4
3
2
1
$
Thewhilecommanddefinesthetestconditiontocheckforeachiteration:
while[$var1-gt0]
As long as the test condition is true, the while command continues to loop through the
commandsdefined.Withinthecommands,thevariableusedinthetestconditionmustbe
modified, or you’ll have an infinite loop. In this example, we use shell arithmetic to
decreasethevariablevaluebyone:
var1=$[$var1-1]
Thewhileloopstopswhenthetestconditionisnolongertrue.
Usingmultipletestcommands
Thewhilecommandallowsyoutodefinemultipletestcommandsonthewhilestatement
line. Only the exit status of the last test command is used to determine when the loop
stops.Thiscancausesomeinterestingresultsifyou’renotcareful.Here’sanexampleof
whatwemean:
$cattest11
#!/bin/bash
#testingamulticommandwhileloop
var1=10
whileecho$var1
[$var1-ge0]
do
echo“Thisisinsidetheloop”
var1=$[$var1-1]
done
$./test11
10
Thisisinsidetheloop
9
Thisisinsidetheloop
8
Thisisinsidetheloop
7
Thisisinsidetheloop
6
Thisisinsidetheloop
5
Thisisinsidetheloop
4
Thisisinsidetheloop
3
Thisisinsidetheloop
2
Thisisinsidetheloop
1
Thisisinsidetheloop
0
Thisisinsidetheloop
-1
$
Paycloseattentiontowhathappenedinthisexample.Twotestcommandsweredefinedin
thewhilestatement:
whileecho$var1
[$var1-ge0]
Thefirsttestsimplydisplaysthecurrentvalueofthe var1variable.Thesecondtestuses
brackets to determine the value of the var1 variable. Inside the loop, an echostatement
displaysasimplemessage,indicatingthattheloopwasprocessed.Noticewhenyourun
theexamplehowtheoutputends:
Thisisinsidetheloop
-1
$
Thewhileloopexecutedtheechostatementwhenthevar1variablewasequaltozeroand
then decreased the var1 variable value. Next, the test commands were executed for the
next iteration. The echo test command was executed, displaying the value of the var1
variable, which is now less than zero. It’s not until the shell executes the test test
commandthatthewhileloopterminates.
This demonstrates that in a multi-command while statement, all the test commands are
executedineachiteration,includingthelastiterationwhenthelasttestcommandfails.Be
careful of this. Another thing to be careful of is how you specify the multiple test
commands.Notethateachtestcommandisonaseparateline!
TheuntilCommand
The until command works in exactly the opposite way from the while command. The
untilcommandrequiresthatyouspecifyatestcommandthatnormallyproducesanonzeroexitstatus.Aslongastheexitstatusofthetestcommandisnon-zero,thebashshell
executes the commands listed in the loop. When the test command returns a zero exit
status,theloopstops.
Asyouwouldexpect,theformatoftheuntilcommandis:
untiltestcommands
do
othercommands
done
Similartothe whilecommand,youcanhavemorethanonetestcommandinthe until
commandstatement.Onlytheexitstatusofthelastcommanddeterminesifthebashshell
executestheothercommandsdefined.
Thefollowingisanexampleofusingtheuntilcommand:
$cattest12
#!/bin/bash
#usingtheuntilcommand
var1=100
until[$var1-eq0]
do
echo$var1
var1=$[$var1-25]
done
$./test12
100
75
50
25
$
Thisexampleteststhe var1 variable to determine when the until loop should stop. As
soonasthevalueofthevariableisequaltozero,the untilcommandstopstheloop.The
same caution as for the while command applies when you use multiple test commands
withtheuntilcommand:
$cattest13
#!/bin/bash
#usingtheuntilcommand
var1=100
untilecho$var1
[$var1-eq0]
do
echoInsidetheloop:$var1
var1=$[$var1-25]
done
$./test13
100
Insidetheloop:100
75
Insidetheloop:75
50
Insidetheloop:50
25
Insidetheloop:25
0
$
Theshellexecutesthetestcommandsspecifiedandstopsonlywhenthelastcommandis
true.
NestingLoops
Aloopstatementcanuseanyothertypeofcommandwithintheloop,includingotherloop
commands.Thisiscalleda nestedloop. Care should be taken when using nested loops,
becauseyou’reperforminganiterationwithinaniteration,whichmultipliesthenumberof
times commands are being run. If you don’t pay close attention to this, it can cause
problemsinyourscripts.
Here’sasimpleexampleofnestingaforloopinsideanotherforloop:
$cattest14
#!/bin/bash
#nestingforloops
for((a=1;a<=3;a++))
do
echo“Startingloop$a:”
for((b=1;b<=3;b++))
do
echo“Insideloop:$b”
done
done
$./test14
Startingloop1:
Insideloop:1
Insideloop:2
Insideloop:3
Startingloop2:
Insideloop:1
Insideloop:2
Insideloop:3
Startingloop3:
Insideloop:1
Insideloop:2
Insideloop:3
$
Thenestedloop(alsocalledtheinnerloop)iteratesthroughitsvaluesforeachiterationof
theouterloop.Noticethatthere’snodifferencebetweenthe doand donecommandsfor
the two loops. The bash shell knows when the first done command is executed that it
referstotheinnerloopandnottheouterloop.
The same applies when you mix loop commands, such as placing a for loop inside a
whileloop:
$cattest15
#!/bin/bash
#placingaforloopinsideawhileloop
var1=5
while[$var1-ge0]
do
echo“Outerloop:$var1”
for((var2=1;$var2<3;var2++))
do
var3=$[$var1*$var2]
echo“Innerloop:$var1*$var2=$var3”
done
var1=$[$var1-1]
done
$./test15
Outerloop:5
Innerloop:5*1=5
Innerloop:5*2=10
Outerloop:4
Innerloop:4*1=4
Innerloop:4*2=8
Outerloop:3
Innerloop:3*1=3
Innerloop:3*2=6
Outerloop:2
Innerloop:2*1=2
Innerloop:2*2=4
Outerloop:1
Innerloop:1*1=1
Innerloop:1*2=2
Outerloop:0
Innerloop:0*1=0
Innerloop:0*2=0
$
Again,theshelldistinguishedbetweenthe doand donecommandsoftheinner forloop
fromthesamecommandsintheouterwhileloop.
Ifyoureallywanttotestyourbrain,youcanevencombineuntilandwhileloops:
$cattest16
#!/bin/bash
#usinguntilandwhileloops
var1=3
until[$var1-eq0]
do
echo“Outerloop:$var1”
var2=1
while[$var2-lt5]
do
var3=$(echo“scale=4;$var1/$var2”|bc)
echo“Innerloop:$var1/$var2=$var3”
var2=$[$var2+1]
done
var1=$[$var1-1]
done
$./test16
Outerloop:3
Innerloop:3/1=3.0000
Innerloop:3/2=1.5000
Innerloop:3/3=1.0000
Innerloop:3/4=.7500
Outerloop:2
Innerloop:2/1=2.0000
Innerloop:2/2=1.0000
Innerloop:2/3=.6666
Innerloop:2/4=.5000
Outerloop:1
Innerloop:1/1=1.0000
Innerloop:1/2=.5000
Innerloop:1/3=.3333
Innerloop:1/4=.2500
$
Theouter untilloopstartswithavalueof3andcontinuesuntilthevalueequals0.The
innerwhileloopstartswithavalueof1andcontinuesaslongasthevalueislessthan5.
Each loop must change the value used in the test condition, or the loop will get stuck
infinitely.
LoopingonFileData
Often,youmustiteratethroughitemsstoredinsideafile.Thisrequirescombiningtwoof
thetechniquescovered:
Usingnestedloops
ChangingtheIFSenvironmentvariable
BychangingtheIFSenvironmentvariable,youcanforcetheforcommandtohandleeach
line in the file as a separate item for processing, even if the data contains spaces. After
you’veextractedanindividuallineinthefile,youmayhavetoloopagaintoextractdata
containedwithinit.
Theclassicexampleofthisisprocessingdatainthe /etc/passwdfile.Thisrequiresthat
you iterate through the /etc/passwd file line by line and then change the IFS variable
valuetoacolonsoyoucanseparatetheindividualcomponentsineachline.
Thefollowingisanexampleofdoingjustthat:
#!/bin/bash
#changingtheIFSvalue
IFS.OLD=$IFS
IFS=$’\n’
forentryin$(cat/etc/passwd)
do
echo“Valuesin$entry–”
IFS=:
forvaluein$entry
do
echo“$value”
done
done
$
Thisscriptusestwodifferent IFSvaluestoparsethedata.Thefirst IFSvalueparsesthe
individuallinesinthe/etc/passwdfile.Theinnerforloopnextchangesthe IFSvalueto
thecolon,whichallowsyoutoparsetheindividualvalueswithinthe/etc/passwdlines.
Whenyourunthisscript,yougetoutputsomethinglikethis:
Valuesinrich:x:501:501:RichBlum:/home/rich:/bin/bashrich
x
501
501
RichBlum
/home/rich
/bin/bash
Valuesinkatie:x:502:502:KatieBlum:/home/katie:/bin/bashkatie
x
506
509
KatieBlum
/home/katie
/bin/bash
Theinnerloopparseseachindividualvalueinthe /etc/passwdentry.Thisisalsoagreat
waytoprocesscomma-separateddata,acommonwaytoimportspreadsheetdata.
ControllingtheLoop
You might be tempted to think that after you start a loop, you’re stuck until the loop
finishes all its iterations. This is not true. A couple of commands help us control what
happensinsideofaloop:
Thebreakcommand
Thecontinuecommand
Eachcommandhasadifferentuseinhowtocontroltheoperationofaloop.Thefollowing
sectionsdescribehowyoucanusethesecommandstocontroltheoperationofyourloops.
Thebreakcommand
Thebreakcommandisasimplewaytoescapealoopinprogress.Youcanusethe break
commandtoexitanytypeofloop,includingwhileanduntilloops.
Youcanusethe breakcommandinseveralsituations.Thissectionshowseachofthese
methods.
Breakingoutofasingleloop
When the shell executes a break command, it attempts to break out of the loop that’s
currentlyprocessing:
$cattest17
#!/bin/bash
#breakingoutofaforloop
forvar1in12345678910
do
if[$var1-eq5]
then
break
fi
echo“Iterationnumber:$var1”
done
echo“Theforloopiscompleted”
$./test17
Iterationnumber:1
Iterationnumber:2
Iterationnumber:3
Iterationnumber:4
Theforloopiscompleted
$
The for loop should normally have iterated through all the values specified in the list.
However, when the if-then condition was satisfied, the shell executed the break
command,whichstoppedtheforloop.
Thistechniquealsoworksforwhileanduntilloops:
$cattest18
#!/bin/bash
#breakingoutofawhileloop
var1=1
while[$var1-lt10]
do
if[$var1-eq5]
then
break
fi
echo“Iteration:$var1”
var1=$[$var1+1]
done
echo“Thewhileloopiscompleted”
$./test18
Iteration:1
Iteration:2
Iteration:3
Iteration:4
Thewhileloopiscompleted
$
The while loop terminated when the if-then condition was met, executing the break
command.
Breakingoutofaninnerloop
Whenyou’reworkingwithmultipleloops,the breakcommandautomaticallyterminates
theinnermostloopyou’rein:
$cattest19
#!/bin/bash
#breakingoutofaninnerloop
for((a=1;a<4;a++))
do
echo“Outerloop:$a”
for((b=1;b<100;b++))
do
if[$b-eq5]
then
break
fi
echo“Innerloop:$b”
done
done
$./test19
Outerloop:1
Innerloop:1
Innerloop:2
Innerloop:3
Innerloop:4
Outerloop:2
Innerloop:1
Innerloop:2
Innerloop:3
Innerloop:4
Outerloop:3
Innerloop:1
Innerloop:2
Innerloop:3
Innerloop:4
$
Theforstatementintheinnerloopspecifiestoiterateuntilthebvariableisequalto100.
However,theif-thenstatementintheinnerloopspecifiesthatwhenthe bvariablevalue
isequalto5,the breakcommandisexecuted.Noticethateventhoughtheinnerloopis
terminatedwiththebreakcommand,theouterloopcontinuesworkingasspecified.
Breakingoutofanouterloop
There may be times when you’re in an inner loop but need to stop the outer loop. The
breakcommandincludesasinglecommandlineparametervalue:
breakn
wheren indicates the level of the loop to break out of. By default, n is 1, indicating to
breakoutofthecurrentloop.Ifyousetntoavalueof2,the breakcommandstopsthe
nextleveloftheouterloop:
$cattest20
#!/bin/bash
#breakingoutofanouterloop
for((a=1;a<4;a++))
do
echo“Outerloop:$a”
for((b=1;b<100;b++))
do
if[$b-gt4]
then
break2
fi
echo“Innerloop:$b”
done
done
$./test20
Outerloop:1
Innerloop:1
Innerloop:2
Innerloop:3
Innerloop:4
$
Nowwhentheshellexecutesthebreakcommand,theouterloopstops.
Thecontinuecommand
The continue command is a way to prematurely stop processing commands inside of a
loopbutnotterminatetheloopcompletely.Thisallowsyoutosetconditionswithinaloop
wheretheshellwon’texecutecommands.Here’sasimpleexampleofusingthecontinue
commandinaforloop:
$cattest21
#!/bin/bash
#usingthecontinuecommand
for((var1=1;var1<15;var1++))
do
if[$var1-gt5]&&[$var1-lt10]
then
continue
fi
echo“Iterationnumber:$var1”
done
$./test21
Iterationnumber:1
Iterationnumber:2
Iterationnumber:3
Iterationnumber:4
Iterationnumber:5
Iterationnumber:10
Iterationnumber:11
Iterationnumber:12
Iterationnumber:13
Iterationnumber:14
$
Whentheconditionsoftheif-thenstatementaremet(thevalueisgreaterthan5andless
than10),theshellexecutesthecontinuecommand,whichskipstherestofthecommands
intheloop,butkeepstheloopgoing.Whentheif-thenconditionisnolongermet,things
returntonormal.
Youcanusethe continuecommandin whileand untilloops,butbeextremelycareful
withwhatyou’redoing.Rememberthatwhentheshellexecutesthe continuecommand,
it skips the remaining commands. If you’re incrementing your test condition variable in
oneofthoseconditions,badthingshappen:
$catbadtest3
#!/bin/bash
#improperlyusingthecontinuecommandinawhileloop
var1=0
whileecho“whileiteration:$var1”
[$var1-lt15]
do
if[$var1-gt5]&&[$var1-lt10]
then
continue
fi
echo“Insideiterationnumber:$var1”
var1=$[$var1+1]
done
$./badtest3|more
whileiteration:0
Insideiterationnumber:0
whileiteration:1
Insideiterationnumber:1
whileiteration:2
Insideiterationnumber:2
whileiteration:3
Insideiterationnumber:3
whileiteration:4
Insideiterationnumber:4
whileiteration:5
Insideiterationnumber:5
whileiteration:6
whileiteration:6
whileiteration:6
whileiteration:6
whileiteration:6
whileiteration:6
whileiteration:6
whileiteration:6
whileiteration:6
whileiteration:6
whileiteration:6
$
You’ll want to make sure you redirect the output of this script to the morecommandso
youcanstopthings.Everythingseemstobegoingjustfineuntilthe if-thenconditionis
met,andtheshellexecutesthecontinuecommand.Whentheshellexecutesthecontinue
command,itskipstheremainingcommandsinthewhileloop.Unfortunately,that’swhere
the $var1countervariablethatistestedinthe whiletestcommandisincremented.That
meansthatthevariableisn’tincremented,asyoucanseefromthecontinuallydisplaying
output.
Aswiththe breakcommand,the continuecommandallowsyoutospecifywhatlevelof
looptocontinuewithacommandlineparameter:
continuen
wherendefinestheloopleveltocontinue.Here’sanexampleofcontinuinganouter for
loop:
$cattest22
#!/bin/bash
#continuinganouterloop
for((a=1;a<=5;a++))
do
echo“Iteration$a:”
for((b=1;b<3;b++))
do
if[$a-gt2]&&[$a-lt4]
then
continue2
fi
var3=$[$a*$b]
echo“Theresultof$a*$bis$var3”
done
done
$./test22
Iteration1:
Theresultof1*1is1
Theresultof1*2is2
Iteration2:
Theresultof2*1is2
Theresultof2*2is4
Iteration3:
Iteration4:
Theresultof4*1is4
Theresultof4*2is8
Iteration5:
Theresultof5*1is5
Theresultof5*2is10
$
Theif-thenstatement:
if[$a-gt2]&&[$a-lt4]
then
continue2
fi
usesthecontinuecommandtostopprocessingthecommandsinsidetheloopbutcontinue
theouterloop.Noticeinthescriptoutputthattheiterationforthevalue3doesn’tprocess
anyinnerloopstatements,becausethe continuecommandstoppedtheprocessing,butit
continueswiththeouterloopprocessing.
ProcessingtheOutputofaLoop
Finally,youcaneitherpipeorredirecttheoutputofaloopwithinyourshellscript.Youdo
thisbyaddingtheprocessingcommandtotheendofthedonecommand:
forfilein/home/rich/*
do
if[-d“$file”]
then
echo“$fileisadirectory”
elif
echo“$fileisafile”
fi
done>output.txt
Insteadofdisplayingtheresultsonthemonitor,theshellredirectstheresultsofthe for
commandtothefileoutput.txt.
Considerthefollowingexampleofredirectingtheoutputofaforcommandtoafile:
$cattest23
#!/bin/bash
#redirectingtheforoutputtoafile
for((a=1;a<10;a++))
do
echo“Thenumberis$a”
done>test23.txt
echo“Thecommandisfinished.”
$./test23
Thecommandisfinished.
$cattest23.txt
Thenumberis1
Thenumberis2
Thenumberis3
Thenumberis4
Thenumberis5
Thenumberis6
Thenumberis7
Thenumberis8
Thenumberis9
$
Theshellcreatesthefiletest23.txtandredirectstheoutputoftheforcommandonlyto
thefile.Theshelldisplaystheechostatementaftertheforcommandjustasnormal.
Thissametechniquealsoworksforpipingtheoutputofalooptoanothercommand:
$cattest24
#!/bin/bash
#pipingalooptoanothercommand
forstatein“NorthDakota”ConnecticutIllinoisAlabamaTennessee
do
echo“$stateisthenextplacetogo”
done|sort
echo“Thiscompletesourtravels”
$./test24
Alabamaisthenextplacetogo
Connecticutisthenextplacetogo
Illinoisisthenextplacetogo
NorthDakotaisthenextplacetogo
Tennesseeisthenextplacetogo
Thiscompletesourtravels
$
Thestatevaluesaren’tlistedinanyparticularorderinthe forcommandlist.Theoutput
ofthe forcommandispipedtothe sortcommand,whichchangestheorderofthe for
command output. Running the script indeed shows that the output was properly sorted
withinthescript.
PracticalExamples
Nowthatyou’veseenhowtousethedifferentwaystocreateloopsinshellscripts,let’s
lookatsomepracticalexamplesofhowtousethem.Loopingisacommonwaytoiterate
throughdataonthesystem,whetherit’sfilesinfoldersordatacontainedinafile.Hereare
acoupleofexamplesthatdemonstrateusingsimpleloopstoworkwithdata.
Findingexecutablefiles
Whenyourunaprogramfromthecommandline,theLinuxsystemsearchesaseriesof
folderslookingforthatfile.ThosefoldersaredefinedinthePATHenvironmentvariable.If
youwanttofindoutjustwhatexecutablefilesareavailableonyoursystemforyoutouse,
justscanallthefoldersinthe PATHenvironmentvariable.Thatmaytakesometimetodo
manually,butit’sabreezeworkingoutasmallshellscripttodothat.
The first step is to create a for loop to iterate through the folders stored in the PATH
environmentvariable.Whenyoudothat,don’tforgettosettheIFSseparatorcharacter:
IFS=:
forfolderin$PATH
do
Nowthatyouhavetheindividualfoldersinthe$foldervariable,youcanuseanotherfor
looptoiteratethroughallthefilesinsidethatparticularfolder:
forfilein$folder/*
do
Thelaststepistocheckwhethertheindividualfileshavetheexecutablepermissionset,
whichyoucandousingtheif-thentestfeature:
if[-x$file]
then
echo“$file”
fi
Andthereyouhaveit!Puttingallthepiecestogetherintoascriptlookslikethis:
$cattest25
#!/bin/bash
#findingfilesinthePATH
IFS=:
forfolderin$PATH
do
echo“$folder:”
forfilein$folder/*
do
if[-x$file]
then
echo“$file”
fi
done
done
$
Whenyourunthecode,yougetalistingoftheexecutablefilesthatyoucanusefromthe
commandline:
$./test25|more
/usr/local/bin:
/usr/bin:
/usr/bin/Mail
/usr/bin/Thunar
/usr/bin/X
/usr/bin/Xorg
/usr/bin/[
/usr/bin/a2p
/usr/bin/abiword
/usr/bin/ac
/usr/bin/activation-client
/usr/bin/addr2line
…
The output shows all the executable files found in all the folders defined in the PATH
environmentvariable,whichisquiteafew!
Creatingmultipleuseraccounts
Thegoalofshellscriptsistomakelifeeasierforthesystemadministrator.Ifyouhappen
toworkinanenvironmentwithlotsofusers,oneofthemostboringtaskscanbecreating
new user accounts. Fortunately, you can use the while loop to make your job a little
easier!
Insteadofhavingtomanuallyenter useraddcommandsforeverynewuseraccountyou
needtocreate,youcanplacethenewuseraccountsinatextfileandcreateasimpleshell
scripttodothatworkforyou.Theformatofthetextfilethatwe’lluselookslikethis:
userid,username
Thefirstentryistheuseridyouwanttouseforthenewuseraccount.Thesecondentryis
thefullnameoftheuser.Thetwovaluesareseparatedbyacomma,makingthisacommaseparatedfileformat,or.csv.Thisisaverycommonfileformatusedinspreadsheets,so
you can easily create the user account list in a spreadsheet program and save it in .csv
formatforyourshellscripttoreadandprocess.
Toreadthefiledata,we’regoingtousealittleshellscriptingtrick.We’llactuallysetthe
IFSseparatorcharactertoacommaasthetestpartofthe whilestatement.Thentoread
theindividuallines,we’llusethereadcommand.Thatlookslikethis:
whileIFS=’,’read–ruseridname
Thereadcommanddoestheworkofmovingontothenextlineoftextinthe.csvtextfile,
so we don’t need another loop to do that. The while command exits when the read
commandreturnsa FALSE value, which happens when it runs out of lines to read in the
file.Tricky!
Tofeedthedatafromthefileintothe whilecommand,youjustusearedirectionsymbol
attheendofthewhilecommand.
Puttingeverythingtogetherresultsinthisscript:
$cattest26
#!/bin/bash
#processnewuseraccounts
input=“users.csv”
whileIFS=’,’read-ruseridname
do
echo“adding$userid”
useradd-c“$name”-m$userid
done<“$input”
$
The $inputvariablepointstothedatafileandisusedastheredirectdataforthe while
command.Theusers.csvfilelookslikethis:
$catusers.csv
rich,RichardBlum
christine,ChristineBresnahan
barbara,BarbaraBlum
tim,TimothyBresnahan
$
To run the problem, you must be the root user account, because the useraddcommand
requiresrootprivileges:
#./test26
addingrich
addingchristine
addingbarbara
addingtim
#
Thenbytakingaquicklookatthe /etc/passwdfile,youcanseethattheaccountshave
beencreated:
#tail/etc/passwd
rich:x:1001:1001:RichardBlum:/home/rich:/bin/bash
christine:x:1002:1002:ChristineBresnahan:/home/christine:/bin/bash
barbara:x:1003:1003:BarbaraBlum:/home/barbara:/bin/bash
tim:x:1004:1004:TimothyBresnahan:/home/tim:/bin/bash
#
Congratulations,you’vesavedyourselflotsoftimeinaddinguseraccounts!
Summary
Looping is an integral part of programming. The bash shell provides three looping
commandsthatyoucanuseinyourscripts.
Theforcommandallowsyoutoiteratethroughalistofvalues,eithersuppliedwithinthe
commandline,containedinavariable,orobtainedbyusingfileglobbing,toextractfile
anddirectorynamesfromawildcardcharacter.
The while command provides a method to loop based on the condition of a command,
usingeitherordinarycommandsorthetestcommand,whichallowsyoutotestconditions
ofvariables.Aslongasthecommand(orcondition)producesazeroexitstatus,thewhile
loopcontinuestoiteratethroughthespecifiedsetofcommands.
Theuntilcommandalsoprovidesamethodtoiteratethroughcommands,butitbasesits
iterations on a command (or condition) producing a non-zero exit status. This feature
allowsyoutosetaconditionthatmustbemetbeforetheiterationstops.
Youcancombineloopsinshellscripts,producingmultiplelayersofloops.Thebashshell
provides the continue and break commands, which allow you to alter the flow of the
normalloopprocessbasedondifferentvalueswithintheloop.
Thebashshellalsoallowsyoutousestandardcommandredirectionandpipingtoalterthe
outputofaloop.Youcanuseredirectiontoredirecttheoutputofalooptoafileorpiping
to redirect the output of a loop to another command. This provides a wealth of features
withwhichyoucancontrolyourshellscriptexecution.
Thenextchapterdiscusseshowtointeractwithyourshellscriptuser.Often,shellscripts
aren’t completely self-contained. They require some sort of external data that must be
supplied at the time you run them. The next chapter discusses different methods with
whichyoucanprovidereal-timedatatoyourshellscriptsforprocessing.
Chapter14
HandlingUserInput
InThisChapter
1. Passingparameters
2. Trackingparameters
3. Beingshifty
4. Workingwithoptions
5. Standardizingoptions
6. Gettinguserinput
Sofaryou’veseenhowtowritescriptsthatinteractwithdata,variables,andfileson
theLinuxsystem.Sometimes,youneedtowriteascriptthathastointeractwiththe
personrunningthescript.Thebashshellprovidesafewdifferentmethodsfor
retrievingdatafrompeople,includingcommandlineparameters(datavaluesadded
afterthecommand),commandlineoptions(single-lettervaluesthatmodifythe
behaviorofthecommand),andthecapabilitytoreadinputdirectlyfromthe
keyboard.Thischapterdiscusseshowtoincorporatethesedifferentmethodsinto
yourbashshellscriptstoobtaindatafromthepersonrunningyourscript.
PassingParameters
The most basic method of passing data to your shell script is to use command line
parameters.Commandlineparametersallowyoutoadddatavaluestothecommandline
whenyouexecutethescript:
$./addem1030
Thisexamplepassestwocommandlineparameters(10and 30)tothescript addem.The
script handles the command line parameters using special variables. The following
sectionsdescribehowtousecommandlineparametersinyourbashshellscripts.
Readingparameters
The bash shell assigns special variables, called positional parameters, to all of the
command line parameters entered. This includes the name of the script the shell is
executing. The positional parameter variables are standard numbers, with $0 being the
script’sname,$1beingthefirstparameter,$2beingthesecondparameter,andsoon,upto
$9fortheninthparameter.
Here’sasimpleexampleofusingonecommandlineparameterinashellscript:
$cattest1.sh
#!/bin/bash
#usingonecommandlineparameter
#
factorial=1
for((number=1;number<=$1;number++))
do
factorial=$[$factorial*$number]
done
echoThefactorialof$1is$factorial
$
$./test1.sh5
Thefactorialof5is120
$
Youcanusethe $1variablejustlikeanyothervariableintheshellscript.Theshellscript
automatically assigns the value from the command line parameter to the variable; you
don’tneedtodoanythingwithit.
Ifyouneedtoentermorecommandlineparameters,eachparametermustbeseparatedby
aspaceonthecommandline:
$cattest2.sh
#!/bin/bash
#testingtwocommandlineparameters
#
total=$[$1*$2]
echoThefirstparameteris$1.
echoThesecondparameteris$2.
echoThetotalvalueis$total.
$
$./test2.sh25
Thefirstparameteris2.
Thesecondparameteris5.
Thetotalvalueis10.
$
Theshellassignseachparametertotheappropriatevariable.
Intheprecedingexample,thecommandlineparametersusedwerebothnumericalvalues.
Youcanalsousetextstringsinthecommandline:
$cattest3.sh
#!/bin/bash
#testingstringparameters
#
echoHello$1,gladtomeetyou.
$
$./test3.shRich
HelloRich,gladtomeetyou.
$
The shell passes the string value entered into the command line to the script. However,
you’llhaveaproblemifyoutrytodothiswithatextstringthatcontainsspaces:
$./test3.shRichBlum
HelloRich,gladtomeetyou.
$
Rememberthateachoftheparametersisseparatedbyaspace,sotheshellinterpretedthe
spaceasjustseparatingthetwovalues.Toincludeaspaceasaparametervalue,youmust
usequotationmarks(eithersingleordoublequotationmarks):
$./test3.sh‘RichBlum’
HelloRichBlum,gladtomeetyou.
$
$./test3.sh“RichBlum”
HelloRichBlum,gladtomeetyou.
$
Note
Thequotationmarksusedwhenyoupasstextstringsasparametersarenotpartofthe
data.Theyjustdelineatethebeginningandtheendofthedata.
Ifyourscriptneedsmorethanninecommandlineparameters,youcancontinue,butthe
variablenameschangeslightly.Aftertheninthvariable,youmustusebracesaroundthe
variablenumber,suchas${10}.Here’sanexampleofdoingthat:
$cattest4.sh
#!/bin/bash
#handlinglotsofparameters
#
total=$[${10}*${11}]
echoThetenthparameteris${10}
echoTheeleventhparameteris${11}
echoThetotalis$total
$
$./test4.sh123456789101112
Thetenthparameteris10
Theeleventhparameteris11
Thetotalis110
$
Thistechniqueallowsyoutoaddasmanycommandlineparameterstoyourscriptsasyou
couldpossiblyneed.
Readingthescriptname
You can use the $0 parameter to determine the script name the shell started from the
commandline.Thiscancomeinhandyifyou’rewritingautilitythatcanhavemultiple
functions.
$cattest5.sh
#!/bin/bash
#Testingthe$0parameter
#
echoThezeroparameterissetto:$0
#
$
$bashtest5.sh
Thezeroparameterissetto:test5.sh
$
However,thereisapotentialproblem.Whenusingadifferentcommandtoruntheshell
script,thecommandbecomesentangledwiththescriptnameinthe$0parameter:
$./test5.sh
Thezeroparameterissetto:./test5.sh
$
Thereisanotherpotentialproblem.Whentheactualstringpassedisthefullscriptpath,
andnotjustthescript’sname,the$0variablegetssettothefullscriptpathandname:
$bash/home/Christine/test5.sh
Thezeroparameterissetto:/home/Christine/test5.sh
$
If you want to write a script that performs different functions based on just the script’s
name,you’llhavetodoalittlework.Youneedtobeabletostripoffwhateverpathisused
torunthescript.Also,youneedtobeabletoremoveanyentangledcommandsfromthe
script.
Fortunately, there’s a handy little command available that does just that. The basename
commandreturnsjustthescript’snamewithoutthepath:
$cattest5b.sh
#!/bin/bash
#Usingbasenamewiththe$0parameter
#
name=$(basename$0)
echo
echoThescriptnameis:$name
#
$bash/home/Christine/test5b.sh
Thescriptnameis:test5b.sh
$
$./test5b.sh
Thescriptnameis:test5b.sh
$
Nowthat’smuchbetter.Youcanusethistechniquetowritescriptsthatperformdifferent
functionsbasedonthescriptnameused.Here’sasimpleexample:
$cattest6.sh
#!/bin/bash
#TestingaMulti-functionscript
#
name=$(basename$0)
#
if[$name=“addem”]
then
total=$[$1+$2]
#
elif[$name=“multem”]
then
total=$[$1*$2]
fi
#
echo
echoThecalculatedvalueis$total
#
$
$cptest6.shaddem
$chmodu+xaddem
$
$ln-stest6.shmultem
$
$ls-l*em
-rwxrw-r—.1ChristineChristine224Jun3023:50addem
lrwxrwxrwx.1ChristineChristine8Jun3023:50multem->test6.sh
$
$./addem25
Thecalculatedvalueis7
$
$./multem25
Thecalculatedvalueis10
$
Theexamplecreatestwoseparatefilenamesfromthetest6.shscript,onebyjustcopying
thefiletoanewscript(addem)andtheotherbyusingasymboliclink(seeChapter3)to
createthenewscript(multem).Inbothcases,thescriptdeterminesthescript’sbasename
andperformstheappropriatefunctionbasedonthatvalue.
Testingparameters
Becarefulwhenusingcommandlineparametersinyourshellscripts.Ifthescriptisrun
withouttheparameters,badthingscanhappen:
$./addem2
./addem:line8:2+:syntaxerror:operandexpected(error
tokenis”“)
Thecalculatedvalueis
$
Whenthescriptassumesthereisdatainaparametervariable,andnodataispresent,most
likely you’ll get an error message from your script. This is a poor way to write scripts.
Alwayscheckyourparameterstomakesurethedataistherebeforeusingit:
$cattest7.sh
#!/bin/bash
#testingparametersbeforeuse
#
if[-n“$1”]
then
echoHello$1,gladtomeetyou.
else
echo“Sorry,youdidnotidentifyyourself.“
fi
$
$./test7.shRich
HelloRich,gladtomeetyou.
$
$./test7.sh
Sorry,youdidnotidentifyyourself.
$
Inthisexample,the -ntestevaluationwasusedtocheckfordatainthe $1commandline
parameter. In the next section, you’ll learn another way to check command line
parameters.
UsingSpecialParameterVariables
Afewspecialbashshellvariablestrackcommandlineparameters.Thissectiondescribes
whattheyareandhowtousethem.
Countingparameters
Asyousawinthelastsection,youshouldverifycommandlineparametersbeforeusing
theminyourscript.Forscriptsthatusemultiplecommandlineparameters,thischecking
cangettedious.
Instead of testing each parameter, you can count how many parameters were entered on
thecommandline.Thebashshellprovidesaspecialvariableforthispurpose.
Thespecial $#variablecontainsthenumberofcommandlineparametersincludedwhen
the script was run. You can use this special variable anywhere in the script, just like a
normalvariable:
$cattest8.sh
#!/bin/bash
#gettingthenumberofparameters
#
echoTherewere$#parameterssupplied.
$
$./test8.sh
Therewere0parameterssupplied.
$
$./test8.sh12345
Therewere5parameterssupplied.
$
$./test8.sh12345678910
Therewere10parameterssupplied.
$
$./test8.sh“RichBlum”
Therewere1parameterssupplied.
$
Now you have the ability to test the number of parameters present before trying to use
them:
$cattest9.sh
#!/bin/bash
#Testingparameters
#
if[$#-ne2]
then
echo
echoUsage:test9.shab
echo
else
total=$[$1+$2]
echo
echoThetotalis$total
echo
fi
#
$
$bashtest9.sh
Usage:test9.shab
$bashtest9.sh10
Usage:test9.shab
$bashtest9.sh1015
Thetotalis25
$
Theif-thenstatementusesthe-neevaluationtoperformanumerictestofthecommand
line parameters supplied. If the correct number of parameters isn’t present, an error
messagedisplaysshowingthecorrectusageofthescript.
Thisvariablealsoprovidesacoolwayofgrabbingthelastparameteronthecommandline
without having to know how many parameters were used. However, you need to use a
littletricktogetthere.
Ifyouthinkthisthrough,youmightthinkthatbecausethe $#variablecontainsthevalue
ofthenumberofparameters,usingthevariable ${$#}wouldrepresentthelastcommand
lineparametervariable.Trythatandseewhathappens:
$catbadtest1.sh
#!/bin/bash
#testinggrabbinglastparameter
#
echoThelastparameterwas${$#}
$
$./badtest1.sh10
Thelastparameterwas15354
$
Wow,whathappened?Obviously,somethingwentwrong.Itturnsoutthatyoucan’tuse
the dollar sign within the braces. Instead, you must replace the dollar sign with an
exclamationmark.Odd,butitworks:
$cattest10.sh
#!/bin/bash
#Grabbingthelastparameter
#
params=$#
echo
echoThelastparameteris$params
echoThelastparameteris${!#}
echo
#
$
$bashtest10.sh12345
Thelastparameteris5
Thelastparameteris5
$
$bashtest10.sh
Thelastparameteris0
Thelastparameteristest10.sh
$
Perfect. This script also assigned the $# variable value to the variable params and then
used that variable within the special command line parameter variable format as well.
Both versions worked. It’s also important to notice that, when there weren’t any
parameters on the command line, the $# value was zero, which is what appears in the
paramsvariable,butthe${!#}variablereturnsthescriptnameusedonthecommandline.
Grabbingallthedata
In some situations you want to grab all the parameters provided on the command line.
Insteadofhavingtomesswithusingthe $#variabletodeterminehowmanyparameters
areonthecommandlineandhavingtoloopthroughallofthem,youcanuseacoupleof
otherspecialvariables.
The$*and$@variablesprovideeasyaccesstoallyourparameters.Bothofthesevariables
includeallthecommandlineparameterswithinasinglevariable.
The $*variabletakesalltheparameterssuppliedonthecommandlineasasingleword.
The word contains each of the values as they appear on the command line. Basically,
insteadoftreatingtheparametersasmultipleobjects,the$*variabletreatsthemallasone
parameter.
The$@variable,ontheotherhand,takesalltheparameterssuppliedonthecommandline
asseparatewordsinthesamestring.Itallowsyoutoiteratethroughthevalues,separating
outeachparametersupplied.Thisismostoftenaccomplishedusingtheforcommand.
Itcaneasilygetconfusingtofigureouthowthesetwovariablesoperate.Let’slookatthe
differencebetweenthetwo:
$cattest11.sh
#!/bin/bash
#testing$*and$@
#
echo
echo“Usingthe\$*method:$*”
echo
echo“Usingthe\$@method:$@”
$
$./test11.shrichbarbarakatiejessica
Usingthe$*method:richbarbarakatiejessica
Usingthe$@method:richbarbarakatiejessica
$
Notice that on the surface, both variables produce the same output, showing all the
commandlineparametersprovidedatonce.
Thefollowingexampledemonstrateswherethedifferencesare:
$cattest12.sh
#!/bin/bash
#testing$*and$@
#
echo
count=1
#
forparamin“$*”
do
echo“\$*Parameter#$count=$param”
count=$[$count+1]
done
#
echo
count=1
#
forparamin“$@”
do
echo“\$@Parameter#$count=$param”
count=$[$count+1]
done
$
$./test12.shrichbarbarakatiejessica
$*Parameter#1=richbarbarakatiejessica
$@Parameter#1=rich
$@Parameter#2=barbara
$@Parameter#3=katie
$@Parameter#4=jessica
$
Nowwe’regettingsomewhere.Byusingthe forcommandtoiteratethroughthespecial
variables,youcanseehowtheyeachtreatthecommandlineparametersdifferently.The
$*variabletreatedalltheparametersasasingleparameter,whilethe $@variabletreated
eachparameterseparately.Thisisagreatwaytoiteratethroughcommandlineparameters.
BeingShifty
Anothertoolyouhaveinyourbashshelltoolbeltisthe shiftcommand.Thebashshell
provides the shift command to help you manipulate command line parameters. The
shiftcommandliterallyshiftsthecommandlineparametersintheirrelativepositions.
Whenyouusethe shiftcommand,itmoveseachparametervariableonepositiontothe
leftbydefault.Thus,thevalueforvariable $3ismovedto $2,thevalueforvariable $2is
movedto$1,andthevalueforvariable$1isdiscarded(notethatthevalueforvariable$0,
theprogramname,remainsunchanged).
This is another great way to iterate through command line parameters, especially if you
don’t know how many parameters are available. You can just operate on the first
parameter,shifttheparametersover,andthenoperateonthefirstparameteragain.
Here’sashortdemonstrationofhowthisworks:
$cattest13.sh
#!/bin/bash
#demonstratingtheshiftcommand
echo
count=1
while[-n“$1”]
do
echo“Parameter#$count=$1”
count=$[$count+1]
shift
done
$
$./test13.shrichbarbarakatiejessica
Parameter#1=rich
Parameter#2=barbara
Parameter#3=katie
Parameter#4=jessica
$
Thescriptperformsa whileloop,testingthelengthofthefirstparameter’svalue.When
the first parameter’s length is zero, the loop ends. After testing the first parameter, the
shiftcommandisusedtoshiftalltheparametersoneposition.
Tip
Becarefulwhenworkingwiththeshiftcommand.Whenaparameterisshiftedout,
itsvalueislostandcan’tberecovered.
Alternatively, you can perform a multiple location shift by providing a parameter to the
shiftcommand.Justprovidethenumberofplacesyouwanttoshift:
$cattest14.sh
#!/bin/bash
#demonstratingamulti-positionshift
#
echo
echo“Theoriginalparameters:$*”
shift2
echo“Here’sthenewfirstparameter:$1”
$
$./test14.sh12345
Theoriginalparameters:12345
Here’sthenewfirstparameter:3
$
By using values in the shift command, you can easily skip over parameters you don’t
need.
WorkingwithOptions
If you’ve been following along in the book, you’ve seen several bash commands that
provide both parameters and options. Options are single letters preceded by a dash that
alter the behavior of a command. This section shows three methods for working with
optionsinyourshellscripts.
Findingyouroptions
Onthesurface,there’snothingallthatspecialaboutcommandlineoptions.Theyappear
on the command line immediately after the script name, just the same as command line
parameters.Infact,ifyouwant,youcanprocesscommandlineoptionsthesamewayyou
processcommandlineparameters.
Processingsimpleoptions
In the test13.sh script earlier, you saw how to use the shift command to work your
way down the command line parameters provided with the script program. You can use
thissametechniquetoprocesscommandlineoptions.
As you extract each individual parameter, use the case statement (see Chapter 12) to
determinewhenaparameterisformattedasanoption:
$cattest15.sh
#!/bin/bash
#extractingcommandlineoptionsasparameters
#
echo
while[-n“$1”]
do
case“$1”in
-a)echo“Foundthe-aoption”;;
-b)echo“Foundthe-boption”;;
-c)echo“Foundthe-coption”;;
*)echo“$1isnotanoption”;;
esac
shift
done
$
$./test15.sh-a-b-c-d
Foundthe-aoption
Foundthe-boption
Foundthe-coption
-disnotanoption
$
The case statement checks each parameter for valid options. When one is found, the
appropriatecommandsareruninthecasestatement.
This method works, no matter in what order the options are presented on the command
line:
$./test15.sh-d-c-a
-disnotanoption
Foundthe-coption
Foundthe-aoption
$
Thecasestatementprocesseseachoptionasitfindsitinthecommandlineparameters.If
anyotherparametersareincludedonthecommandline,youcanincludecommandsinthe
catch-allpartofthecasestatementtoprocessthem.
Separatingoptionsfromparameters
Oftenyou’llrunintosituationswhereyou’llwanttousebothoptionsandparametersfora
shell script. The standard way to do this in Linux is to separate the two with a special
character code that tells the script when the options are finished and when the normal
parametersstart.
ForLinux,thisspecialcharacteristhedoubledash(—).Theshellusesthedoubledashto
indicate the end of the option list. After seeing the double dash, your script can safely
processtheremainingcommandlineparametersasparametersandnotoptions.
Tocheckforthedoubledash,simplyaddanotherentryinthecasestatement:
$cattest16.sh
#!/bin/bash
#extractingoptionsandparameters
echo
while[-n“$1”]
do
case“$1”in
-a)echo“Foundthe-aoption”;;
-b)echo“Foundthe-boption”;;
-c)echo“Foundthe-coption”;;
—)shift
break;;
*)echo“$1isnotanoption”;;
esac
shift
done
#
count=1
forparamin$@
do
echo“Parameter#$count:$param”
count=$[$count+1]
done
$
Thisscriptusesthebreakcommandtobreakoutofthewhileloopwhenitencountersthe
doubledash.Becausewe’rebreakingoutprematurely,weneedtoensurethatwestickin
anothershiftcommandtogetthedoubledashoutoftheparametervariables.
Forthefirsttest,tryrunningthescriptusinganormalsetofoptionsandparameters:
$./test16.sh-c-a-btest1test2test3
Foundthe-coption
Foundthe-aoption
Foundthe-boption
test1isnotanoption
test2isnotanoption
test3isnotanoption
$
The results show that the script assumed that all the command line parameters were
optionswhenitprocessedthem.Next,trythesamething,onlythistimeusingthedouble
dashtoseparatetheoptionsfromtheparametersonthecommandline:
$./test16.sh-c-a-b—test1test2test3
Foundthe-coption
Foundthe-aoption
Foundthe-boption
Parameter#1:test1
Parameter#2:test2
Parameter#3:test3
$
Whenthescriptreachesthedoubledash,itstopsprocessingoptionsandassumesthatany
remainingparametersarecommandlineparameters.
Processingoptionswithvalues
Someoptionsrequireanadditionalparametervalue.Inthesesituations,thecommandline
lookssomethinglikethis:
$./testing.sh-atest1-b-c-dtest2
Yourscriptmustbeabletodetectwhenyourcommandlineoptionrequiresanadditional
parameterandbeabletoprocessitappropriately.Here’sanexampleofhowtodothat:
$cattest17.sh
#!/bin/bash
#extractingcommandlineoptionsandvalues
echo
while[-n“$1”]
do
case“$1”in
-a)echo“Foundthe-aoption”;;
-b)param=”$2”
echo“Foundthe-boption,withparametervalue$param”
shift;;
-c)echo“Foundthe-coption”;;
—)shift
break;;
*)echo“$1isnotanoption”;;
esac
shift
done
#
count=1
forparamin“$@”
do
echo“Parameter#$count:$param”
count=$[$count+1]
done
$
$./test17.sh-a-btest1-d
Foundthe-aoption
Foundthe-boption,withparametervaluetest1
-disnotanoption
$
Inthisexample,the casestatementdefinesthreeoptionsthatitprocesses.The -boption
alsorequiresanadditionalparametervalue.Becausetheparameterbeingprocessedis $1,
youknowthattheadditionalparametervalueislocatedin $2(becausealltheparameters
areshiftedaftertheyareprocessed).Justextracttheparametervaluefromthe$2variable.
Ofcourse,becauseweusedtwoparameterspotsforthisoption,youalsoneedtosetthe
shiftcommandtoshiftoneadditionalposition.
Just as with the basic feature, this process works no matter what order you place the
options in (just remember to include the appropriate option parameter with the each
option):
$./test17.sh-btest1-a-d
Foundthe-boption,withparametervaluetest1
Foundthe-aoption
-disnotanoption
$
Nowyouhavethebasicabilitytoprocesscommandlineoptionsinyourshellscripts,but
there are limitations. For example, this doesn’t work if you try to combine multiple
optionsinoneparameter:
$./test17.sh-ac
-acisnotanoption
$
ItisacommonpracticeinLinuxtocombineoptions,andifyourscriptisgoingtobeuserfriendly, you’ll want to offer this feature for your users as well. Fortunately, there’s
anothermethodforprocessingoptionsthatcanhelpyou.
Usingthegetoptcommand
The getopt command is a great tool to have handy when processing command line
optionsandparameters.Itreorganizesthecommandlineparameterstomakeparsingthem
inyourscripteasier.
Lookingatthecommandformat
The getopt command can take a list of command line options and parameters, in any
form,andautomaticallyturnthemintotheproperformat.Itusesthefollowingcommand
format:
getoptoptstringparameters
Theoptstringisthekeytotheprocess.Itdefinesthevalidoptionlettersthatcanbeused
inthecommandline.Italsodefineswhichoptionlettersrequireaparametervalue.
First, list each command line option letter you’re going to use in your script in the
optstring.Thenplaceacolonaftereachoptionletterthatrequiresaparametervalue.The
getoptcommandparsesthesuppliedparametersbasedontheoptstringyoudefine.
Tip
Amoreadvancedversionofthegetoptcommand,calledgetopts(noticeitis
plural),isavailable.Thegetoptscommandiscoveredlaterinthischapter.Because
oftheirnearlyidenticalspelling,it’seasytogetthesetwocommandsconfused.Be
careful!
Here’sasimpleexampleofhowgetoptworks:
$getoptab:cd-a-btest1-cdtest2test3
-a-btest1-c-d—test2test3
$
Theoptstringdefinesfourvalidoptionletters,a,b,c,andd.Acolon(:)isplacedbehind
the letter b in order to require option b to have a parameter value. When the getopt
commandruns,itexaminestheprovidedparameterlist(-a-btest1-cdtest2test3)
andparsesitbasedonthesuppliedoptstring.Noticethatitautomaticallyseparatedthecdoptionsintotwoseparateoptionsandinsertedthedoubledashtoseparatetheadditional
parametersontheline.
Ifyouspecifyaparameteroptionnotinthe optstring,bydefaultthe getoptcommand
producesanerrormessage:
$getoptab:cd-a-btest1-cdetest2test3
getopt:invalidoption—e
-a-btest1-c-d—test2test3
$
Ifyouprefertojustignoretheerrormessages,usegetoptwiththe-qoption:
$getopt-qab:cd-a-btest1-cdetest2test3
-a-b‘test1’-c-d—‘test2’‘test3’
$
Note that the getopt command options must be listed before the optstring. Now you
shouldbereadytousethiscommandinyourscriptstoprocesscommandlineoptions.
Usinggetoptinyourscripts
Youcanusethe getoptcommandinyourscriptstoformatanycommandlineoptionsor
parametersenteredforyourscript.It’salittletricky,however,touse.
The trick is to replace the existing command line options and parameters with the
formattedversionproducedbythe getoptcommand.Thewaytodothatistousethe set
command.
YousawthesetcommandbackinChapter6.Thesetcommandworkswiththedifferent
variablesintheshell.
Oneofthe setcommandoptionsisthedoubledash(—).Thedoubledashinstructs setto
replace the command line parameter variables with the values on the set command’s
commandline.
The trick then is to feed the original script command line parameters to the getopt
commandandthenfeedtheoutputofthegetoptcommandtothesetcommandtoreplace
theoriginalcommandlineparameterswiththenicelyformattedonesfrom getopt.This
lookssomethinglikethis:
set—$(getopt-qab:cd“$@”)
Now the values of the original command line parameter variables are replaced with the
outputfromthegetoptcommand,whichformatsthecommandlineparametersforus.
Usingthistechnique,wecannowwritescriptsthathandleourcommandlineparameters
forus:
$cattest18.sh
#!/bin/bash
#Extractcommandlineoptions&valueswithgetopt
#
set—$(getopt-qab:cd“$@”)
#
echo
while[-n“$1”]
do
case“$1”in
-a)echo“Foundthe-aoption”;;
-b)param=”$2”
echo“Foundthe-boption,withparametervalue$param”
shift;;
-c)echo“Foundthe-coption”;;
—)shift
break;;
*)echo“$1isnotanoption”;;
esac
shift
done
#
count=1
forparamin“$@”
do
echo“Parameter#$count:$param”
count=$[$count+1]
done
#
$
You’llnoticethisisbasicallythesamescriptasintest17.sh.Theonlythingthatchanged
istheadditionofthegetoptcommandtohelpformatourcommandlineparameters.
Nowwhenyourunthescriptwithcomplexoptions,thingsworkmuchbetter:
$./test18.sh-ac
Foundthe-aoption
Foundthe-coption
$
Andofcourse,alltheoriginalfeaturesworkjustfineaswell:
$./test18.sh-a-btest1-cdtest2test3test4
Foundthe-aoption
Foundthe-boption,withparametervalue‘test1’
Foundthe-coption
Parameter#1:‘test2’
Parameter#2:‘test3’
Parameter#3:‘test4’
$
Nowthingsarelookingprettyfancy.However,there’sstillonesmallbugthatlurksinthe
getoptcommand.Checkoutthisexample:
$./test18.sh-a-btest1-cd“test2test3”test4
Foundthe-aoption
Foundthe-boption,withparametervalue‘test1’
Foundthe-coption
Parameter#1:‘test2
Parameter#2:test3’
Parameter#3:‘test4’
$
The getopt command isn’t good at dealing with parameter values with spaces and
quotationmarks.Itinterpretedthespaceastheparameterseparator,insteadoffollowing
thedoublequotationmarksandcombiningthetwovaluesintooneparameter.Fortunately,
thisproblemhasanothersolution.
Advancingtogetopts
The getoptscommand(noticethatitisplural)isbuiltintothebashshell.Itlooksmuch
likeitsgetoptcousin,buthassomeexpandedfeatures.
Unlike getopt, which produces one output for all the processed options and parameters
foundinthecommandline,the getoptscommandworksontheexistingshellparameter
variablessequentially.
It processes the parameters it detects in the command line one at a time each time it’s
called.Whenitrunsoutofparameters,itexitswithanexitstatusgreaterthanzero.This
makesitgreatforusinginloopstoparsealltheparametersonthecommandline.
Here’stheformatofthegetoptscommand:
getoptsoptstringvariable
The optstring value is similar to the one used in the getopt command. Valid option
letters are listed in the optstring, along with a colon if the option letter requires a
parameter value. To suppress error messages, start the optstring with a colon. The
getoptscommandplacesthecurrentparameterinthe variabledefinedinthecommand
line.
Thegetoptscommandusestwoenvironmentvariables.TheOPTARGenvironmentvariable
contains the value to be used if an option requires a parameter value. The OPTIND
environment variable contains the value of the current location within the parameter list
where getopts left off. This allows you to continue processing other command line
parametersafterfinishingtheoptions.
Let’slookatasimpleexamplethatusesthegetoptscommand:
$cattest19.sh
#!/bin/bash
#simpledemonstrationofthegetoptscommand
#
echo
whilegetopts:ab:copt
do
case“$opt”in
a)echo“Foundthe-aoption”;;
b)echo“Foundthe-boption,withvalue$OPTARG”;;
c)echo“Foundthe-coption”;;
*)echo“Unknownoption:$opt”;;
esac
done
$
$./test19.sh-abtest1-c
Foundthe-aoption
Foundthe-boption,withvaluetest1
Foundthe-coption
$
The while statement defines the getopts command, specifying what command line
optionstolookfor,alongwiththevariablename(opt)tostoretheminforeachiteration.
You’ll notice something different about the case statement in this example. When the
getoptscommandparsesthecommandlineoptions,itstripsofftheleadingdash,soyou
don’tneedleadingdashesinthecasedefinitions.
Thegetoptscommandoffersseveralnicefeatures.Forstarters,youcanincludespacesin
yourparametervalues:
$./test19.sh-b“test1test2”-a
Foundthe-boption,withvaluetest1test2
Foundthe-aoption
$
Anothernicefeatureisthatyoucanruntheoptionletterandtheparametervaluetogether
withoutaspace:
$./test19.sh-abtest1
Foundthe-aoption
Foundthe-boption,withvaluetest1
$
The getoptscommandcorrectlyparsedthe test1valuefromthe -boption.Inaddition,
the getoptscommandbundlesanyundefinedoptionitfindsinthecommandlineintoa
singleoutput,thequestionmark:
$./test19.sh-d
Unknownoption:?
$
$./test19.sh-acde
Foundthe-aoption
Foundthe-coption
Unknownoption:?
Unknownoption:?
$
Any option letter not defined in the optstring value is sent to your code as a question
mark.
The getoptscommandknowswhentostopprocessingoptionsandleavetheparameters
for you to process. As getopts processes each option, it increments the OPTIND
environment variable by one. When you’ve reached the end of the getopts processing,
youcanusetheOPTINDvaluewiththeshiftcommandtomovetotheparameters:
$cattest20.sh
#!/bin/bash
#Processingoptions&parameterswithgetopts
#
echo
whilegetopts:ab:cdopt
do
case“$opt”in
a)echo“Foundthe-aoption”;;
b)echo“Foundthe-boption,withvalue$OPTARG”;;
c)echo“Foundthe-coption”;;
d)echo“Foundthe-doption”;;
*)echo“Unknownoption:$opt”;;
esac
done
#
shift$[$OPTIND-1]
#
echo
count=1
forparamin“$@”
do
echo“Parameter$count:$param”
count=$[$count+1]
done
#
$
$./test20.sh-a-btest1-dtest2test3test4
Foundthe-aoption
Foundthe-boption,withvaluetest1
Foundthe-doption
Parameter1:test2
Parameter2:test3
Parameter3:test4
$
Nowyouhaveafull-featuredcommandlineoptionandparameterprocessingutilityyou
canuseinallyourshellscripts!
StandardizingOptions
When you create your shell script, obviously you’re in control of what happens. It’s
completelyuptoyouastowhichletteroptionsyouselecttouseandhowyouselecttouse
them.
However,afewletteroptionshaveachievedasomewhatstandardmeaningintheLinux
world.Ifyouleveragetheseoptionsinyourshellscript,yourscriptswillbemoreuserfriendly.
Table14.1showssomeofthecommonmeaningsforcommandlineoptionsusedinLinux.
Table14.1CommonLinuxCommandLineOptions
Option
-a
-c
-d
-e
-f
-h
-i
-l
-n
-o
-q
-r
-s
-v
-x
-y
Description
Showsallobjects
Producesacount
Specifiesadirectory
Expandsanobject
Specifiesafiletoreaddatafrom
Displaysahelpmessageforthecommand
Ignorestextcase
Producesalongformatversionoftheoutput
Usesanon-interactive(batch)mode
Specifiesanoutputfiletoredirectalloutputto
Runsinquietmode
Processesdirectoriesandfilesrecursively
Runsinsilentmode
Producesverboseoutput
Excludesanobject
Answersyestoallquestions
You’ll probably recognize most of these option meanings just from working with the
various bash commands throughout the book. Using the same meaning for your options
helpsusersinteractwithyourscriptwithouthavingtoworryaboutmanuals.
GettingUserInput
Althoughprovidingcommandlineoptionsandparametersisagreatwaytogetdatafrom
yourscriptusers,sometimesyourscriptneedstobemoreinteractive.Sometimesyouneed
to ask a question while the script is running and wait for a response from the person
runningyourscript.Thebashshellprovidesthereadcommandjustforthispurpose.
Readingbasics
The readcommandacceptsinputeitherfromstandardinput(suchasfromthekeyboard)
or from another file descriptor. After receiving the input, the read command places the
dataintoavariable.Here’sthereadcommandatitssimplest:
$cattest21.sh
#!/bin/bash
#testingthereadcommand
#
echo-n“Enteryourname:“
readname
echo“Hello$name,welcometomyprogram.“
#
$
$./test21.sh
Enteryourname:RichBlum
HelloRichBlum,welcometomyprogram.
$
That’sprettysimple.Noticethatthe echocommandthatproducedthepromptusesthe -n
option.Thissuppressesthenewlinecharacterattheendofthestring,allowingthescript
usertoenterdataimmediatelyafterthestring,insteadofonthenextline.Thisgivesyour
scriptsamoreform-likeappearance.
Infact,the readcommandincludesthe -poption,whichallowsyoutospecifyaprompt
directlyinthereadcommandline:
$cattest22.sh
#!/bin/bash
#testingtheread-poption
#
read-p“Pleaseenteryourage:”age
days=$[$age*365]
echo“Thatmakesyouover$daysdaysold!“
#
$
$./test22.sh
Pleaseenteryourage:10
Thatmakesyouover3650daysold!
$
You’ll notice in the first example that when a name was entered, the read command
assigned both the first name and last name to the same variable. The read command
assigns all data entered at the prompt to a single variable, or you can specify multiple
variables.Eachdatavalueenteredisassignedtothenextvariableinthelist.Ifthelistof
variablesrunsoutbeforethedatadoes,theremainingdataisassignedtothelastvariable:
$cattest23.sh
#!/bin/bash
#enteringmultiplevariables
#
read-p“Enteryourname:”firstlast
echo“Checkingdatafor$last,$first…”
$
$./test23.sh
Enteryourname:RichBlum
CheckingdataforBlum,Rich…
$
You can also specify no variables on the read command line. If you do that, the read
commandplacesanydataitreceivesinthespecialenvironmentvariableREPLY:
$cattest24.sh
#!/bin/bash
#TestingtheREPLYEnvironmentvariable
#
read-p“Enteryourname:“
echo
echoHello$REPLY,welcometomyprogram.
#
$
$./test24.sh
Enteryourname:Christine
HelloChristine,welcometomyprogram.
$
The REPLY environment variable contains all the data entered in the input, and it can be
usedintheshellscriptasanyothervariable.
Timingout
Becarefulwhenusingthereadcommand.Yourscriptmaygetstuckwaitingforthescript
usertoenterdata.Ifthescriptmustgoonregardlessofwhetheranydatawasentered,you
canusethe-toptiontospecifyatimer.The-toptionspecifiesthenumberofsecondsfor
thereadcommandtowaitforinput.Whenthetimerexpires,thereadcommandreturnsa
non-zeroexitstatus:
$cattest25.sh
#!/bin/bash
#timingthedataentry
#
ifread-t5-p“Pleaseenteryourname:”name
then
echo“Hello$name,welcometomyscript”
else
echo
echo“Sorry,tooslow!“
fi
$
$./test25.sh
Pleaseenteryourname:Rich
HelloRich,welcometomyscript
$
$./test25.sh
Pleaseenteryourname:
Sorry,tooslow!
$
Becausethereadcommandexitswithanon-zeroexitstatusifthetimerexpires,it’seasy
tousethestandardstructuredstatements,suchasanif-thenstatementorawhileloopto
trackwhathappened.Inthisexample,whenthetimerexpires,the ifstatementfails,and
theshellexecutesthecommandsintheelsesection.
Instead of timing the input, you can also set the read command to count the input
characters. When a preset number of characters has been entered, it automatically exits,
assigningtheentereddatatothevariable:
$cattest26.sh
#!/bin/bash
#gettingjustonecharacterofinput
#
read-n1-p“Doyouwanttocontinue[Y/N]?”answer
case$answerin
Y|y)echo
echo“fine,continueon…”;;
N|n)echo
echoOK,goodbye
exit;;
esac
echo“Thisistheendofthescript”
$
$./test26.sh
Doyouwanttocontinue[Y/N]?Y
fine,continueon…
Thisistheendofthescript
$
$./test26.sh
Doyouwanttocontinue[Y/N]?n
OK,goodbye
$
This example uses the -n option with the value of 1, instructing the read command to
acceptonlyasinglecharacterbeforeexiting.Assoonasyoupressthesinglecharacterto
answer,thereadcommandacceptstheinputandpassesittothevariable.Youdon’tneed
topresstheEnterkey.
Readingwithnodisplay
Sometimesyouneedinputfromthescriptuser,butyoudon’twantthatinputtodisplayon
themonitor.Theclassicexampleiswhenenteringpasswords,butthereareplentyofother
typesofdatathatyouneedtohide.
The-soptionpreventsthedataenteredinthereadcommandfrombeingdisplayedonthe
monitor;actually,thedataisdisplayed,butthe readcommandsetsthetextcolortothe
sameasthebackgroundcolor.Here’sanexampleofusingthe-soptioninascript:
$cattest27.sh
#!/bin/bash
#hidinginputdatafromthemonitor
#
read-s-p“Enteryourpassword:”pass
echo
echo“Isyourpasswordreally$pass?“
$
$./test27.sh
Enteryourpassword:
IsyourpasswordreallyT3st1ng?
$
The data typed at the input prompt doesn’t appear on the monitor but is assigned to the
variableforuseinthescript.
Readingfromafile
Finally, you can also use the read command to read data stored in a file on the Linux
system.Eachcalltothe readcommandreadsasinglelineoftextfromthefile.Whenno
morelinesareleftinthefile,thereadcommandexitswithanon-zeroexitstatus.
Thetrickypartisgettingthedatafromthefiletothe readcommand.Themostcommon
methodisto pipetheresultofthe catcommandofthefiledirectlytoa whilecommand
thatcontainsthereadcommand.Here’sanexample:
$cattest28.sh
#!/bin/bash
#readingdatafromafile
#
count=1
cattest|whilereadline
do
echo“Line$count:$line”
count=$[$count+1]
done
echo“Finishedprocessingthefile”
$
$cattest
Thequickbrowndogjumpsoverthelazyfox.
Thisisatest,thisisonlyatest.
ORomeo,Romeo!WhereforeartthouRomeo?
$
$./test28.sh
Line1:Thequickbrowndogjumpsoverthelazyfox.
Line2:Thisisatest,thisisonlyatest.
Line3:ORomeo,Romeo!WhereforeartthouRomeo?
Finishedprocessingthefile
$
The whilecommandloopcontinuesprocessinglinesofthefilewiththe readcommand,
untilthereadcommandexitswithanon-zeroexitstatus.
Summary
Thischaptershowedthreemethodsforretrievingdatafromthescriptuser.Commandline
parameters allow users to enter data directly on the command line when they run the
script.Thescriptusespositionalparameterstoretrievethecommandlineparametersand
assignthemtovariables.
The shiftcommandallowsyoutomanipulatethecommandlineparametersbyrotating
themwithinthepositionalparameters.Thiscommandallowsyoutoeasilyiteratethrough
theparameterswithoutknowinghowmanyparametersareavailable.
You can use three special variables when working with command line parameters. The
shellsetsthe $#variabletothenumberofparametersenteredonthecommandline.The
$*variablecontainsalltheparametersasasinglestring,andthe $@variablecontainsall
the parameters as separate words. These variables come in handy when you’re trying to
processlongparameterlists.
Besidesparameters,yourscriptuserscanusecommandlineoptionstopassinformationto
yourscript.Commandlineoptionsaresinglelettersprecededbyadash.Differentoptions
canbeassignedtoalterthebehaviorofyourscript.
Thebashshellprovidesthreewaystohandlecommandlineoptions.
Thefirstwayistohandlethemjustlikecommandlineparameters.Youcaniteratethrough
theoptionsusingthepositionalparametervariables,processingeachoptionasitappears
onthecommandline.
Another way to handle command line options is with the getopt command. This
commandconvertscommandlineoptionsandparametersintoastandardformatthatyou
can process in your script. The getopt command allows you to specify which letters it
recognizes as options and which options require an additional parameter value. The
getopt command processes the standard command line parameters and outputs the
optionsandparametersintheproperorder.
Thefinalmethodforhandlingcommandlineoptionsisviathe getoptscommand(note
that it’s plural). The getopts command provides more advanced processing of the
command line parameters. It allows for multi-value parameters, along with identifying
optionsnotdefinedbythescript.
An interactive method to obtain data from your script users is the read command. The
read command allows your scripts to query users for information and wait. The read
commandplacesanydataenteredbythescriptuserintooneormorevariables,whichyou
canusewithinthescript.
Severaloptionsareavailableforthe readcommandthatallowyoutocustomizethedata
input into your script, such as using hidden data entry, applying timed data entry, and
requestingaspecificnumberofinputcharacters.
Inthenextchapter,welookfurtherintohowbashshellscriptsoutputdata.Sofar,you’ve
seenhowtodisplaydataonthemonitorandredirectittoafile.Next,weexploreafew
otheroptionsthatyouhaveavailablenotonlytodirectdatatospecificlocationsbutalso
todirectspecifictypesofdatatospecificlocations.Thiswillhelpmakeyourshellscripts
lookprofessional!
Chapter15
PresentingData
InThisChapter
1. Revisitingredirection
2. Standardinputandoutput
3. Reportingerrors
4. Throwingawaydata
5. Creatinglogfiles
Sofarthescriptsshowninthisbookdisplayinformationeitherbyechoingdatatothe
monitororbyredirectingdatatoafile.Chapter11demonstratedhowtoredirectthe
outputofacommandtoafile.Thischapterexpandsonthattopicbyshowingyou
howyoucanredirecttheoutputofyourscripttodifferentlocationsonyourLinux
system.
UnderstandingInputandOutput
Sofar,you’veseentwomethodsfordisplayingtheoutputfromyourscripts:
Displayingoutputonthemonitorscreen
Redirectingoutputtoafile
Both methods produced an all-or-nothing approach to data output. There are times,
however,whenitwouldbenicetodisplaysomedataonthemonitorandotherdataina
file.Fortheseinstances,itcomesinhandytoknowhowLinuxhandlesinputandoutput
soyoucangetyourscriptoutputtotherightplace.
ThefollowingsectionsdescribehowtousethestandardLinuxinputandoutputsystemto
youradvantage,tohelpdirectscriptoutputtospecificlocations.
Standardfiledescriptors
The Linux system handles every object as a file. This includes the input and output
process. Linux identifies each file object using a filedescriptor. The file descriptor is a
non-negative integer that uniquely identifies open files in a session. Each process is
allowedtohaveuptonineopenfiledescriptorsatatime.Thebashshellreservesthefirst
threefiledescriptors(0,1,and2)forspecialpurposes.TheseareshowninTable15.1.
Table15.1LinuxStandardFileDescriptors
FileDescriptor Abbreviation Description
STDIN
0
Standardinput
STDOUT
1
Standardoutput
STDERR
2
Standarderror
Thesethreespecialfiledescriptorshandletheinputandoutputfromyourscript.Theshell
uses them to direct the default input and output in the shell to the appropriate location,
which by default is usually your monitor. The following sections describe each of these
standardfiledescriptorsingreaterdetail.
STDIN
The STDIN file descriptor references the standard input to the shell. For a terminal
interface,thestandardinputisthekeyboard.Theshellreceivesinputfromthekeyboard
ontheSTDINfiledescriptorandprocesseseachcharacterasyoutypeit.
When you use the input redirect symbol (<), Linux replaces the standard input file
descriptor with the file referenced by the redirection. It reads the file and retrieves data
justasifitweretypedonthekeyboard.
Manybashcommandsacceptinputfrom STDIN,especiallyifnofilesarespecifiedonthe
command line. Here’s an example of using the cat command with data entered from
STDIN:
$cat
thisisatest
thisisatest
thisisasecondtest.
thisisasecondtest.
When you enter the cat command on the command line by itself, it accepts input from
STDIN.Asyouentereachline,thecatcommandechoesthelinetothedisplay.
However,youcanalsousethe STDINredirectsymboltoforcethe catcommandtoaccept
inputfromanotherfileotherthanSTDIN:
$cat<testfile
Thisisthefirstline.
Thisisthesecondline.
Thisisthethirdline.
$
Nowthe catcommandusesthelinesthatarecontainedinthe testfilefileastheinput.
You can use this technique to input data to any shell command that accepts data from
STDIN.
STDOUT
The STDOUT file descriptor references the standard output for the shell. On a terminal
interface,thestandardoutputistheterminalmonitor.Alloutputfromtheshell(including
programsandscriptsyourunintheshell)isdirectedtothestandardoutput,whichisthe
monitor.
MostbashcommandsdirecttheiroutputtotheSTDOUTfiledescriptorbydefault.Asshown
inChapter11,youcanchangethatusingoutputredirection:
$ls-l>test2
$cattest2
total20
-rw-rw-r—1richrich532014-10-1611:30test
-rw-rw-r—1richrich02014-10-1611:32test2
-rw-rw-r—1richrich732014-10-1611:23testfile
$
Withtheoutputredirectionsymbol,alltheoutputthatnormallywouldgotothemonitoris
insteadredirectedtothedesignatedredirectionfilebytheshell.
Youcanalsoappenddatatoafile.Youdothisusingthe>>symbol:
$who>>test2
$cattest2
total20
-rw-rw-r—1richrich532014-10-1611:30test
-rw-rw-r—1richrich02014-10-1611:32test2
-rw-rw-r—1richrich732014-10-1611:23testfile
richpts/02014-10-1715:34(192.168.1.2)
$
Theoutputgeneratedbythe whocommandisappendedtothedataalreadyinthe test2
file.
However, if you use the standard output redirection for your scripts, you can run into a
problem.Here’sanexampleofwhatcanhappeninyourscript:
$ls-albadfile>test3
ls:cannotaccessbadfile:Nosuchfileordirectory
$cattest3
$
Whenacommandproducesanerrormessage,theshelldoesn’tredirecttheerrormessage
to the output redirection file. The shell created the output redirection file, but the error
message appeared on the monitor screen. Notice that there isn’t an error when trying to
displaythecontentsofthetest3file.Thetest3filewascreatedjustfine,butit’sempty.
Theshellhandleserrormessagesseparatelyfromthenormaloutput.Ifyou’recreatinga
shell script that runs in background mode, often you must rely on the output messages
being sent to a log file. Using this technique, if any error messages occur, they don’t
appearinthelogfile.Youneedtodosomethingdifferent.
STDERR
Theshellhandleserrormessagesusingthespecial STDERRfiledescriptor.The STDERRfile
descriptorreferencesthestandarderroroutputfortheshell.Thisisthelocationwherethe
shellsendserrormessagesgeneratedbytheshellorprogramsandscriptsrunninginthe
shell.
Bydefault,theSTDERRfiledescriptorpointstothesameplaceastheSTDOUTfiledescriptor
(even though they are assigned different file descriptor values). This means that, by
default,allerrormessagesgotothemonitordisplay.
However,asyousawintheexample,whenyouredirectSTDOUT,thisdoesn’tautomatically
redirect STDERR. When working with scripts, you’ll often want to change that behavior,
especiallyifyou’reinterestedinloggingerrormessagestoalogfile.
Redirectingerrors
You’ve already seen how to redirect the STDOUT data by using the redirection symbol.
Redirectingthe STDERRdataisn’tmuchdifferent;youjustneedtodefinethe STDERRfile
descriptorwhenyouusetheredirectionsymbol.Youcandothisinacoupleofways.
Redirectingerrorsonly
AsyousawinTable15.1,theSTDERRfiledescriptorissettothevalue2.Youcanselectto
redirectonlyerrormessagesbyplacingthisfiledescriptorvalueimmediatelybeforethe
redirectionsymbol.Thevaluemustappearimmediatelybeforetheredirectionsymbolorit
doesn’twork:
$ls-albadfile2>test4
$cattest4
ls:cannotaccessbadfile:Nosuchfileordirectory
$
Now when you run the command, the error message doesn’t appear on the monitor.
Instead,theoutputfilecontainsanyerrormessagesthataregeneratedbythecommand.
Usingthismethod,theshellredirectstheerrormessagesonly,notthenormaldata.Here’s
anotherexampleofmixingSTDOUTandSTDERRmessagesinthesameoutput:
$ls-altestbadtesttest22>test5
-rw-rw-r—1richrich1582014-10-1611:32test2
$cattest5
ls:cannotaccesstest:Nosuchfileordirectory
ls:cannotaccessbadtest:Nosuchfileordirectory
$
The normal STDOUT output from the ls command still goes to the default STDOUT file
descriptor,whichisthemonitor.Becausethecommandredirectsfiledescriptor2output
(STDERR) to an output file, the shell sends any error messages generated directly to the
specifiedredirectionfile.
Redirectingerrorsanddata
Ifyouwanttoredirectbotherrorsandthenormaloutput,youneedtousetworedirection
symbols. You need to precede each with the appropriate file descriptor for the data you
want to redirect and then have them point to the appropriate output file for holding the
data:
$ls-altesttest2test3badtest2>test61>test7
$cattest6
ls:cannotaccesstest:Nosuchfileordirectory
ls:cannotaccessbadtest:Nosuchfileordirectory
$cattest7
-rw-rw-r—1richrich1582014-10-1611:32test2
-rw-rw-r—1richrich02014-10-1611:33test3
$
Theshellredirectsthenormaloutputofthe lscommandthatwouldhavegoneto STDOUT
tothetest7fileusingthe1>symbol.AnyerrormessagesthatwouldhavegonetoSTDERR
wereredirectedtothetest6fileusingthe2>symbol.
Youcanusethistechniquetoseparatenormalscriptoutputfromanyerrormessagesthat
occur in the script. This allows you to easily identify errors without having to wade
throughthousandsoflinesofnormaloutputdata.
Alternatively, if you want, you can redirect both STDERRand STDOUT output to the same
outputfile.Thebashshellprovidesaspecialredirectionsymboljustforthispurpose,the
&>symbol:
$ls-altesttest2test3badtest&>test7
$cattest7
ls:cannotaccesstest:Nosuchfileordirectory
ls:cannotaccessbadtest:Nosuchfileordirectory
-rw-rw-r—1richrich1582014-10-1611:32test2
-rw-rw-r—1richrich02014-10-1611:33test3
$
Whenyouusethe&>symbol,alltheoutputgeneratedbythecommandissenttothesame
location,bothdataanderrors.Noticethatoneoftheerrormessagesisoutoforderfrom
what you’d expect. The error message for the badtest file (the last file to be listed)
appears second in the output file. The bash shell automatically gives error messages a
higher priority than the standard output. This allows you to view the error messages
together,ratherthanscatteredthroughouttheoutputfile.
RedirectingOutputinScripts
Youcanusethe STDOUTand STDERR file descriptors in your scripts to produce output in
multiple locations simply by redirecting the appropriate file descriptors. There are two
methodsforredirectingoutputinthescript:
Temporarilyredirectingeachline
Permanentlyredirectingallcommandsinthescript
Thefollowingsectionsdescribehoweachofthesemethodsworks.
Temporaryredirections
If you want to purposely generate error messages in your script, you can redirect an
individual output line to STDERR. You just need to use the output redirection symbol to
redirecttheoutputtotheSTDERRfiledescriptor.Whenyouredirecttoafiledescriptor,you
mustprecedethefiledescriptornumberwithanampersand(&):
echo“Thisisanerrormessage”>&2
This line displays the text wherever the STDERR file descriptor for the script is pointing,
instead of the normal STDOUT. The following is an example of a script that uses this
feature:
$cattest8
#!/bin/bash
#testingSTDERRmessages
echo“Thisisanerror”>&2
echo“Thisisnormaloutput”
$
Ifyourunthescriptasnormal,youdon’tnoticeanydifference:
$./test8
Thisisanerror
Thisisnormaloutput
$
Rememberthat,bydefault,Linuxdirectsthe STDERRoutputto STDOUT.However,ifyou
redirect STDERR when running the script, any text directed to STDERR in the script is
redirected:
$./test82>test9
Thisisnormaloutput
$cattest9
Thisisanerror
$
Perfect!ThetextdisplayedusingSTDOUTappearsonthemonitor,whiletheechostatement
textsenttoSTDERRisredirectedtotheoutputfile.
Thismethodisgreatforgeneratingerrormessagesinyourscripts.Ifsomeoneusesyour
scripts, they can easily redirect the error messages using the STDERR file descriptor, as
shown.
Permanentredirections
Ifyouhavelotsofdatathatyou’reredirectinginyourscript,itcangettedioushavingto
redirect every echo statement. Instead, you can tell the shell to redirect a specific file
descriptorforthedurationofthescriptbyusingtheexeccommand:
$cattest10
#!/bin/bash
#redirectingalloutputtoafile
exec1>testout
echo“Thisisatestofredirectingalloutput”
echo“fromascripttoanotherfile.”
echo“withouthavingtoredirecteveryindividualline”
$./test10
$cattestout
Thisisatestofredirectingalloutput
fromascripttoanotherfile.
withouthavingtoredirecteveryindividualline
$
TheexeccommandstartsanewshellandredirectstheSTDOUTfiledescriptortoafile.All
outputinthescriptthatgoestoSTDOUTisinsteadredirectedtothefile.
YoucanalsoredirecttheSTDOUTinthemiddleofascript:
$cattest11
#!/bin/bash
#redirectingoutputtodifferentlocations
exec2>testerror
echo“Thisisthestartofthescript”
echo“nowredirectingalloutputtoanotherlocation”
exec1>testout
echo“Thisoutputshouldgotothetestoutfile”
echo“butthisshouldgotothetesterrorfile”>&2
$
$./test11
Thisisthestartofthescript
nowredirectingalloutputtoanotherlocation
$cattestout
Thisoutputshouldgotothetestoutfile
$cattesterror
butthisshouldgotothetesterrorfile
$
The script uses the exec command to redirect any output going to STDERR to the file
testerror.Next,thescriptusestheechostatementtodisplayafewlinestoSTDOUT.After
that,the execcommandisusedagaintoredirect STDOUTtothe testoutfile.Noticethat
evenwhenSTDOUTisredirected,youcanstillspecifytheoutputfromanechostatementto
gotoSTDERR,whichinthiscaseisstillredirectedtothetesterrorfile.
This feature can come in handy when you want to redirect the output of just parts of a
scripttoanalternativelocation,suchasanerrorlog.There’sjustoneproblemyouruninto
whenusingthis.
Afteryouredirect STDOUTor STDERR,youcan’teasilyredirectthembacktotheiroriginal
location.Ifyouneedtoswitchbackandforthwithyourredirection,youneedtolearna
trick. The “Creating Your Own Redirection” section later in this chapter discusses this
trickandhowtouseitinyourshellscripts.
RedirectingInputinScripts
You can use the same technique used to redirect STDOUT and STDERR in your scripts to
redirectSTDINfromthekeyboard.TheexeccommandallowsyoutoredirectSTDINfroma
fileontheLinuxsystem:
exec0<testfile
This command informs the shell that it should retrieve input from the file testfile
instead of STDIN. This redirection applies anytime the script requests input. Here’s an
exampleofthisinaction:
$cattest12
#!/bin/bash
#redirectingfileinput
exec0<testfile
count=1
whilereadline
do
echo“Line#$count:$line”
count=$[$count+1]
done
$./test12
Line#1:Thisisthefirstline.
Line#2:Thisisthesecondline.
Line#3:Thisisthethirdline.
$
Chapter 14 showed you how to use the read command to read data entered from the
keyboardbyauser.ByredirectingSTDINfromafile,whenthereadcommandattemptsto
readfromSTDIN,itretrievesdatafromthefileinsteadofthekeyboard.
This is an excellent technique to read data in files for processing in your scripts. A
commontaskforLinuxsystemadministratorsistoreaddatafromlogfilesforprocessing.
Thisistheeasiestwaytoaccomplishthattask.
CreatingYourOwnRedirection
Whenyouredirectinputandoutputinyourscript,you’renotlimitedtothethreedefault
file descriptors. I mentioned that you could have up to nine open file descriptors in the
shell.Theothersixfiledescriptorsarenumberedfrom3through8andareavailablefor
youtouseaseitherinputoroutputredirection.Youcanassignanyofthesefiledescriptors
toafileandthenusetheminyourscriptsaswell.Thissectionshowsyouhowtousethe
otherfiledescriptorsinyourscripts.
Creatingoutputfiledescriptors
Youassignafiledescriptorforoutputbyusingtheexeccommand.Aswiththestandard
file descriptors, after you assign an alternative file descriptor to a file location, that
redirection stays permanent until you reassign it. Here’s a simple example of using an
alternativefiledescriptorinascript:
$cattest13
#!/bin/bash
#usinganalternativefiledescriptor
exec3>test13out
echo“Thisshoulddisplayonthemonitor”
echo“andthisshouldbestoredinthefile”>&3
echo“Thenthisshouldbebackonthemonitor”
$./test13
Thisshoulddisplayonthemonitor
Thenthisshouldbebackonthemonitor
$cattest13out
andthisshouldbestoredinthefile
$
The script uses the exec command to redirect file descriptor 3 to an alternative file
location. When the script executes the echo statements, they display on STDOUT as you
wouldexpect.However,theechostatementsthatyouredirecttofiledescriptor3gotothe
alternativefile.Thisallowsyoutokeepnormaloutputforthemonitorandredirectspecial
informationtofiles,suchaslogfiles.
Youcanalsousetheexeccommandtoappenddatatoanexistingfileinsteadofcreatinga
newfile:
exec3>>test13out
Nowtheoutputisappendedtothetest13outfileinsteadofcreatinganewfile.
Redirectingfiledescriptors
Here’s the trick to help you bring back a redirected file descriptor. You can assign an
alternativefiledescriptortoastandardfiledescriptor,andviceversa.Thismeansthatyou
can redirect the original location of STDOUT to an alternative file descriptor and then
redirectthatfiledescriptorbackto STDOUT.Thismightsoundsomewhatcomplicated,but
inpracticeit’sfairlystraightforward.Thisexamplewillclearthingsupforyou:
$cattest14
#!/bin/bash
#storingSTDOUT,thencomingbacktoit
exec3>&1
exec1>test14out
echo“Thisshouldstoreintheoutputfile”
echo“alongwiththisline.”
exec1>&3
echo“Nowthingsshouldbebacktonormal”
$
$./test14
Nowthingsshouldbebacktonormal
$cattest14out
Thisshouldstoreintheoutputfile
alongwiththisline.
$
This example is a little crazy so let’s walk through it piece by piece. First, the script
redirectsfiledescriptor3tothecurrentlocationoffiledescriptor1,whichisSTDOUT.This
meansthatanyoutputsenttofiledescriptor3goestothemonitor.
Thesecond execcommandredirects STDOUTtoafile.Theshellnowredirectsanyoutput
sent to STDOUT directly to the output file. However, file descriptor 3 still points to the
originallocationofSTDOUT,whichisthemonitor.Ifyousendoutputdatatofiledescriptor
3atthispoint,itstillgoestothemonitor,eventhoughSTDOUTisredirected.
After sending some output to STDOUT, which points to a file, the script then redirects
STDOUT to the current location of file descriptor 3, which is still set to the monitor. This
meansthatnowSTDOUTpointstoitsoriginallocation,themonitor.
Thismethodcangetconfusing,butit’sacommonwaytotemporarilyredirectoutputin
scriptfilesandthensettheoutputbacktothenormalsettings.
Creatinginputfiledescriptors
You can redirect input file descriptors exactly the same way as output file descriptors.
Savethe STDINfiledescriptorlocationtoanotherfiledescriptorbeforeredirectingittoa
file;whenyou’refinishedreadingthefile,youcanrestoreSTDINtoitsoriginallocation:
$cattest15
#!/bin/bash
#redirectinginputfiledescriptors
exec6<&0
exec0<testfile
count=1
whilereadline
do
echo“Line#$count:$line”
count=$[$count+1]
done
exec0<&6
read-p“Areyoudonenow?”answer
case$answerin
Y|y)echo“Goodbye”;;
N|n)echo“Sorry,thisistheend.”;;
esac
$./test15
Line#1:Thisisthefirstline.
Line#2:Thisisthesecondline.
Line#3:Thisisthethirdline.
Areyoudonenow?y
Goodbye
$
In this example, file descriptor 6 is used to hold the location for STDIN. The script then
redirects STDINtoafile.Alltheinputforthe readcommandcomesfromtheredirected
STDIN,whichisnowtheinputfile.
When all the lines have been read, the script returns STDIN to its original location by
redirectingittofiledescriptor6.ThescriptteststomakesurethatSTDINisbacktonormal
byusinganotherreadcommand,whichthistimewaitsforinputfromthekeyboard.
Creatingaread/writefiledescriptor
As odd as it may seem, you can also open a single file descriptor for both input and
output.Youcanthenusethesamefiledescriptortobothreaddatafromafileandwrite
datatothesamefile.
Youneedtobeespeciallycarefulwiththismethod,however.Asyoureadandwritedata
toandfromafile,theshellmaintainsaninternalpointer,indicatingwhereitisinthefile.
Anyreadingorwritingoccurswherethefilepointerlastleftoff.Thiscanproducesome
interestingresultsifyou’renotcareful.Lookatthisexample:
$cattest16
#!/bin/bash
#testinginput/outputfiledescriptor
exec3<>testfile
readline<&3
echo“Read:$line”
echo“Thisisatestline”>&3
$cattestfile
Thisisthefirstline.
Thisisthesecondline.
Thisisthethirdline.
$./test16
Read:Thisisthefirstline.
$cattestfile
Thisisthefirstline.
Thisisatestline
ine.
Thisisthethirdline.
$
Thisexampleusestheexeccommandtoassignfiledescriptor3forbothinputandoutput
senttoandfromthefiletestfile.Next,itusesthereadcommandtoreadthefirstlinein
the file, using the assigned file descriptor, and then it displays the read line of data in
STDOUT.Afterthat,itusestheechostatementtowritealineofdatatothefileopenedwith
thesamefiledescriptor.
Whenyourunthescript,atfirstthingslookjustfine.Theoutputshowsthatthescriptread
thefirstlineinthe testfilefile.However,ifyoudisplaythecontentsofthe testfile
fileafterrunningthescript,youseethatthedatawrittentothefileoverwrotetheexisting
data.
Whenthescriptwritesdatatothefile,itstartswherethefilepointerislocated.The read
commandreadsthefirstlineofdata,soitleftthefilepointerpointingtothefirstcharacter
inthesecondlineofdata.Whenthe echostatementoutputsdatatothefile,itplacesthe
dataatthecurrentlocationofthefilepointer,overwritingwhateverdatawasthere.
Closingfiledescriptors
Ifyoucreatenewinputoroutputfiledescriptors,theshellautomaticallyclosesthemwhen
the script exits. There are situations, however, when you need to manually close a file
descriptorbeforetheendofthescript.
Tocloseafiledescriptor,redirectittothespecialsymbol&-.Thisishowthislooksinthe
script:
exec3>&-
This statement closes file descriptor 3, preventing it from being used any more in the
script.Here’sanexampleofwhathappenswhenyoutrytouseaclosedfiledescriptor:
$catbadtest
#!/bin/bash
#testingclosingfiledescriptors
exec3>test17file
echo“Thisisatestlineofdata”>&3
exec3>&echo“Thiswon’twork”>&3
$./badtest
./badtest:3:Badfiledescriptor
$
Afteryouclosethefiledescriptor,youcan’twriteanydatatoitinyourscriptortheshell
producesanerrormessage.
There’s yet another thing to be careful of when closing file descriptors. If you open the
sameoutputfilelateroninyourscript,theshellreplacestheexistingfilewithanewfile.
This means that if you output any data, it overwrites the existing file. Consider the
followingexampleofthisproblem:
$cattest17
#!/bin/bash
#testingclosingfiledescriptors
exec3>test17file
echo“Thisisatestlineofdata”>&3
exec3>&cattest17file
exec3>test17file
echo“This’llbebad”>&3
$./test17
Thisisatestlineofdata
$cattest17file
This’llbebad
$
Aftersendingadatastringtothetest17filefileandclosingthefiledescriptor,thescript
usesthecatcommandtodisplaythecontentsofthefile.Sofar,sogood.Next,thescript
reopenstheoutputfileandsendsanotherdatastringtoit.Whenyoudisplaythecontents
of the output file, all you see is the second data string. The shell overwrote the original
outputfile.
ListingOpenFileDescriptors
With only nine file descriptors available to you, you’d think that it wouldn’t be hard to
keepthingsstraight.Sometimes,however,it’seasytogetlostwhentryingtokeeptrackof
which file descriptor is redirected where. To help you keep your sanity, the bash shell
providesthelsofcommand.
The lsofcommandlistsalltheopenfiledescriptorsontheentireLinuxsystem.Thisis
somewhatofacontroversialfeature,becauseitcanprovideinformationabouttheLinux
systemtonon-system-administrators.That’swhymanyLinuxsystemshidethiscommand
sousersdon’taccidentallystumbleacrossit.
OnmanyLinuxsystems(suchasFedora)the lsofcommandislocatedinthe /usr/sbin
directory.Torunitwithanormaluseraccount,Ihavetoreferenceitbyitsfullpathname:
$/usr/sbin/lsof
This produces an amazing amount of output. It displays information about every file
currently open on the Linux system. This includes all the processes running on
background,aswellasanyuseraccountsloggedintothesystem.
Plenty of command line parameters and options are available to help filter out the lsof
output.Themostcommonlyusedare -p,whichallowsyoutospecifyaprocessID(PID),
and-d,whichallowsyoutospecifythefiledescriptornumberstodisplay.
ToeasilydeterminethecurrentPIDoftheprocess,youcanusethespecialenvironment
variable $$, which the shell sets to the current PID. The -a option is used to perform a
BooleanANDoftheresultsoftheothertwooptions,toproducethefollowing:
$/usr/sbin/lsof-a-p$$-d0,1,2
COMMANDPIDUSERFDTYPEDEVICESIZENODENAME
bash3344rich0uCHR136,02/dev/pts/0
bash3344rich1uCHR136,02/dev/pts/0
bash3344rich2uCHR136,02/dev/pts/0
$
Thisshowsthedefaultfiledescriptors(0,1,and2)forthecurrentprocess(thebashshell).
The default output of lsof contains several columns of information, described in Table
15.2.
Table15.2DefaultlsofOutput
Column
COMMAND
PID
USER
FD
TYPE
Description
Thefirstninecharactersofthenameofthecommandintheprocess
TheprocessIDoftheprocess
Theloginnameoftheuserwhoownstheprocess
Thefiledescriptornumberandaccesstype[r—(read),w—(write),u—
(read/write)]
Thetypeoffile[CHR—(character),BLK—(block),DIR—(directory),
REG—(regularfile)]
DEVICE
SIZE
NODE
NAME
Thedevicenumbers(majorandminor)ofthedevice
Ifavailable,thesizeofthefile
Thenodenumberofthelocalfile
Thenameofthefile
Thefiletypeassociatedwith STDIN, STDOUT,and STDERRischaractermode.Becausethe
STDIN,STDOUT,andSTDERRfiledescriptorsallpointtotheterminal,thenameoftheoutput
file is the device name of the terminal. All three standard files are available for both
readingandwriting(althoughitdoesseemoddtobeabletowritetoSTDINandreadfrom
STDOUT).
Now, let’s look at the results of the lsof command from inside a script that’s opened a
coupleofalternativefiledescriptors:
$cattest18
#!/bin/bash
#testinglsofwithfiledescriptors
exec3>test18file1
exec6>test18file2
exec7<testfile
/usr/sbin/lsof-a-p$$-d0,1,2,3,6,7
$./test18
COMMANDPIDUSERFDTYPEDEVICESIZENODENAME
test183594rich0uCHR136,02/dev/pts/0
test183594rich1uCHR136,02/dev/pts/0
test183594rich2uCHR136,02/dev/pts/0
183594rich3wREG253,00360712/home/rich/test18file1
183594rich6wREG253,00360715/home/rich/test18file2
183594rich7rREG253,073360717/home/rich/testfile
$
Thescriptcreatesthreealternativefiledescriptors,twoforoutput(3and6)andonefor
input(7).Whenthescriptrunsthelsofcommand,youcanseethenewfiledescriptorsin
the output. We truncated the first part of the output so you could see the results of the
filename. The filename shows the complete pathname for the files used in the file
descriptors. It shows each of the files as type REG, which indicates that they are regular
filesonthefilesystem.
SuppressingCommandOutput
Sometimes,youmaynotwanttodisplayanyoutputfromyourscript.Thisoftenoccursif
you’rerunningascriptasabackgroundprocess(seeChapter16).Ifanyerrormessages
occurfromthescriptwhileit’srunninginthebackground,theshelle-mailsthemtothe
owner of the process. This can get tedious, especially if you run scripts that generate
minornuisanceerrors.
To solve that problem, you can redirect STDERR to a special file called the nullfile.The
nullfileisprettymuchwhatitsaysitis—afilethatcontainsnothing.Anydatathatthe
shelloutputstothenullfileisnotsaved,thusthedataarelost.
The standard location for the null file on Linux systems is /dev/null. Any data you
redirecttothatlocationisthrownawayanddoesn’tappear:
$ls-al>/dev/null
$cat/dev/null
$
Thisisacommonwaytosuppressanyerrormessageswithoutactuallysavingthem:
$ls-albadfiletest162>/dev/null
-rwxr—r—1richrich135Oct2919:57test16*
$
You can also use the /dev/null file for input redirection as an input file. Because the
/dev/nullfilecontainsnothing,itisoftenusedbyprogrammerstoquicklyremovedata
fromanexistingfilewithouthavingtoremovethefileandre-createit:
$cattestfile
Thisisthefirstline.
Thisisthesecondline.
Thisisthethirdline.
$cat/dev/null>testfile
$cattestfile
$
Thefiletestfilestillexistsonthesystem,butnowitisempty.Thisisacommonmethod
usedtoclearoutlogfilesthatmustremaininplaceforapplicationstooperate.
UsingTemporaryFiles
TheLinuxsystemcontainsaspecialdirectorylocationreservedfortemporaryfiles.Linux
uses the /tmp directory for files that don’t need to be kept indefinitely. Most Linux
distributionsconfigurethesystemtoautomaticallyremoveanyfilesinthe /tmpdirectory
atbootup.
Anyuseraccountonthesystemhasprivilegestoreadandwritefilesinthe/tmpdirectory.
This feature provides an easy way for you to create temporary files that you don’t
necessarilyhavetoworryaboutcleaningup.
There’s even a specific command to use for creating a temporary file. The mktemp
commandallowsyoutoeasilycreateauniquetemporaryfileinthe /tmpfolder.Theshell
createsthefilebutdoesn’tuseyourdefaultumaskvalue(seeChapter7).Instead,itonly
assignsreadandwritepermissionstothefile’sownerandmakesyoutheownerofthefile.
Afteryoucreatethefile,youhavefullaccesstoreadandwritetoandfromitfromyour
script,butnooneelsecanaccessit(otherthantherootuser,ofcourse).
Creatingalocaltemporaryfile
Bydefault,mktempcreatesafileinthelocaldirectory.Tocreateatemporaryfileinalocal
directory with the mktemp command, you just need to specify a filename template. The
templateconsistsofanytextfilename,plussixX’sappendedtotheendofthefilename:
$mktemptesting.XXXXXX
$ls-altesting*
-rw––-1richrich0Oct1721:30testing.UfIi13
$
ThemktempcommandreplacesthesixX’swithasix-charactercodetoensurethefilename
isuniqueinthedirectory.Youcancreatemultipletemporaryfilesandbeassuredthateach
oneisunique:
$mktemptesting.XXXXXX
testing.1DRLuV
$mktemptesting.XXXXXX
testing.lVBtkW
$mktemptesting.XXXXXX
testing.PgqNKG
$ls-ltesting*
-rw––-1richrich0Oct1721:57testing.1DRLuV
-rw––-1richrich0Oct1721:57testing.PgqNKG
-rw––-1richrich0Oct1721:30testing.UfIi13
-rw––-1richrich0Oct1721:57testing.lVBtkW
$
Asyoucansee,theoutputofthe mktempcommandisthenameofthefilethatitcreates.
When you use the mktemp command in a script, you’ll want to save that filename in a
variable,soyoucanrefertoitlateroninthescript:
$cattest19
#!/bin/bash
#creatingandusingatempfile
tempfile=$(mktemptest19.XXXXXX)
exec3>$tempfile
echo“Thisscriptwritestotempfile$tempfile”
echo“Thisisthefirstline”>&3
echo“Thisisthesecondline.”>&3
echo“Thisisthelastline.”>&3
exec3>&echo“Donecreatingtempfile.Thecontentsare:”
cat$tempfile
rm-f$tempfile2>/dev/null
$./test19
Thisscriptwritestotempfiletest19.vCHoya
Donecreatingtempfile.Thecontentsare:
Thisisthefirstline
Thisisthesecondline.
Thisisthelastline.
$ls-altest19*
-rwxr—r—1richrich356Oct2922:03test19*
$
Thescriptusesthemktempcommandtocreateatemporaryfileandassignsthefilenameto
the$tempfilevariable.Itthenusesthetemporaryfileastheoutputredirectionfileforfile
descriptor3.Afterdisplayingthetemporaryfilenameon STDOUT,itwritesafewlinesto
thetemporaryfile,andthenitclosesthefiledescriptor.Finally,itdisplaysthecontentsof
thetemporaryfileandthenusesthermcommandtoremoveit.
Creatingatemporaryfilein/tmp
The -toptionforces mktemp to create the file in the temporary directory of the system.
Whenyouusethisfeature,the mktempcommandreturnsthefullpathnameusedtocreate
thetemporaryfile,notjustthefilename:
$mktemp-ttest.XXXXXX
/tmp/test.xG3374
$ls-al/tmp/test*
-rw––-1richrich02014-10-2918:41/tmp/test.xG3374
$
Because the mktemp command returns the full pathname, you can then reference the
temporary file from any directory on the Linux system, no matter where it places the
temporarydirectory:
$cattest20
#!/bin/bash
#creatingatempfilein/tmp
tempfile=$(mktemp-ttmp.XXXXXX)
echo“Thisisatestfile.”>$tempfile
echo“Thisisthesecondlineofthetest.”>>$tempfile
echo“Thetempfileislocatedat:$tempfile”
cat$tempfile
rm-f$tempfile
$./test20
Thetempfileislocatedat:/tmp/tmp.Ma3390
Thisisatestfile.
Thisisthesecondlineofthetest.
$
When mktempcreatesthetemporaryfile,itreturnsthefullpathnametotheenvironment
variable.Youcanthenusethatvalueinanycommandtoreferencethetemporaryfile.
Creatingatemporarydirectory
The -doptiontellsthe mktempcommandtocreateatemporarydirectoryinsteadofafile.
You can then use that directory for whatever purposes you need, such as creating
additionaltemporaryfiles:
$cattest21
#!/bin/bash
#usingatemporarydirectory
tempdir=$(mktemp-ddir.XXXXXX)
cd$tempdir
tempfile1=$(mktemptemp.XXXXXX)
tempfile2=$(mktemptemp.XXXXXX)
exec7>$tempfile1
exec8>$tempfile2
echo“Sendingdatatodirectory$tempdir”
echo“Thisisatestlineofdatafor$tempfile1”>&7
echo“Thisisatestlineofdatafor$tempfile2”>&8
$./test21
Sendingdatatodirectorydir.ouT8S8
$ls-al
total72
drwxr-xr-x3richrich4096Oct1722:20./
drwxr-xr-x9richrich4096Oct1709:44../
drwx––2richrich4096Oct1722:20dir.ouT8S8/
-rwxr—r—1richrich338Oct1722:20test21*
$cddir.ouT8S8
[dir.ouT8S8]$ls-al
total16
drwx––2richrich4096Oct1722:20./
drwxr-xr-x3richrich4096Oct1722:20../
-rw––-1richrich44Oct1722:20temp.N5F3O6
-rw––-1richrich44Oct1722:20temp.SQslb7
[dir.ouT8S8]$cattemp.N5F3O6
Thisisatestlineofdatafortemp.N5F3O6
[dir.ouT8S8]$cattemp.SQslb7
Thisisatestlineofdatafortemp.SQslb7
[dir.ouT8S8]$
Thescriptcreatesadirectoryinthecurrentdirectoryandusesthe cdcommandtochange
to that directory before creating two temporary files. The two temporary files are then
assignedtofiledescriptorsandusedtostoreoutputfromthescript.
LoggingMessages
Sometimes, it’s beneficial to send output both to the monitor and to a file for logging.
Insteadofhavingtoredirectoutputtwice,youcanusethespecialteecommand.
The tee command is like a T-connector for pipes. It sends data from STDIN to two
destinations at the same time. One destination is STDOUT. The other destination is a
filenamespecifiedontheteecommandline:
teefilename
Because teeredirectsdatafrom STDIN,youcanuseitwiththepipecommandtoredirect
outputfromanycommand:
$date|teetestfile
SunOct1918:56:21EDT2014
$cattestfile
SunOct1918:56:21EDT2014
$
Theoutputappearsin STDOUTandiswrittentothefilespecified.Becareful:Bydefault,
theteecommandoverwritestheoutputfileoneachuse:
$who|teetestfile
richpts/02014-10-1718:41(192.168.1.2)
$cattestfile
richpts/02014-10-1718:41(192.168.1.2)
$
Ifyouwanttoappenddatatothefile,youmustusethe-aoption:
$date|tee-atestfile
SunOct1918:58:05EDT2014
$cattestfile
richpts/02014-10-1718:41(192.168.1.2)
SunOct1918:58:05EDT2014
$
Usingthistechnique,youcanbothsavedatainfilesanddisplaythedataonthemonitor
foryourusers:
$cattest22
#!/bin/bash
#usingtheteecommandforlogging
tempfile=test22file
echo“Thisisthestartofthetest”|tee$tempfile
echo“Thisisthesecondlineofthetest”|tee-a$tempfile
echo“Thisistheendofthetest”|tee-a$tempfile
$./test22
Thisisthestartofthetest
Thisisthesecondlineofthetest
Thisistheendofthetest
$cattest22file
Thisisthestartofthetest
Thisisthesecondlineofthetest
Thisistheendofthetest
$
Nowyoucansaveapermanentcopyofyouroutputatthesametimeasyou’redisplaying
ittoyourusers.
PracticalExample
Fileredirectionisverycommonbothwhenreadingfilesintoscriptsandwhenoutputting
datafromascriptintoafile.Thisexamplescriptdoesbothofthosethings.Itreadsa.csvformatteddatafileandoutputs SQL INSERT statements to insert the data into a database
(seeChapter25).
Theshellscriptusesacommandlineparametertodefinethenameofthe .csvfilefrom
whichtoreadthedata.The .csvformatisusedtoexportdatafromspreadsheets,soyou
canplacethedatabasedataintoaspreadsheet,savethespreadsheetin .csvformat,read
thefile,andcreateINSERTstatementstoinsertthedataintoaMySQLdatabase.
Here’swhatthescriptlookslike:
$cattest23
#!/bin/bash
#readfileandcreateINSERTstatementsforMySQL
outfile=‘members.sql’
IFS=’,’
whilereadlnamefnameaddresscitystatezip
do
cat>>$outfile<<EOF
INSERTINTOmembers(lname,fname,address,city,state,zip)VALUES
(‘$lname’,‘$fname’,‘$address’,‘$city’,‘$state’,‘$zip’);
EOF
done<${1}
$
That’s a pretty short script, thanks to the file redirection that goes on! There are three
redirection operations happening in the script. The while loop uses the read statement
(discussedinChapter14)toreadtextfromthedatafile.Noticeinthe donestatementthe
redirectionsymbol:
done<${1}
The $1 represents the first command line parameter when you run the test23program.
Thatspecifiesthedatafilefromwhichtoreadthedata.Thereadstatementparsesthetext
usingtheIFScharacter,whichwespecifyasacomma.
Theothertworedirectionoperationsinthescriptbothappearinthesamestatement:
cat>>$outfile<<EOF
This one statement has one output append redirection (the double greater-than symbol)
and one input append redirection (the double less-than symbol). The output redirection
appendsthecatcommandoutputtothefilespecifiedbythe$outfilevariable.Theinput
tothecatcommandisredirectedfromthestandardinputtousethedatastoredinsidethe
script.TheEOFsymbolmarksthestartandenddelimiterofthedatathat’sappendedto
thefile:
INSERTINTOmembers(lname,fname,address,city,state,zip)VALUES
(‘$lname’,‘$fname’,‘$address’,‘$city’,‘$state’,‘$zip’);
ThetextcreatesastandardSQLINSERTstatement.Noticethatthedatavaluesarereplaced
withthevariablesforthedatareadfromthereadstatement.
Sobasicallythe whileloopreadsonthedataonelineatatime,plugsthosedatavalues
intotheINSERTstatementtemplate,thenoutputstheresulttotheoutputfile.
Forthisexperiment,Iusedthisastheinputdatafile:
$catmembers.csv
Blum,Richard,123MainSt.,Chicago,IL,60601
Blum,Barbara,123MainSt.,Chicago,IL,60601
Bresnahan,Christine,456OakAve.,Columbus,OH,43201
Bresnahan,Timothy,456OakAve.,Columbus,OH,43201
$
Whenyourunthescript,nothingappearsintheoutputonthemonitor:
$./test23<members.csv
$
Butwhenyoulookatthemembers.sqloutputfile,youshouldseetheoutputdata:
$catmembers.sql
INSERTINTOmembers(lname,fname,address,city,state,zip)VALUES(‘Blum’,
‘Richard’,‘123MainSt.’,‘Chicago’,‘IL’,‘60601’);
INSERTINTOmembers(lname,fname,address,city,state,zip)VALUES(‘Blum’,
‘Barbara’,‘123MainSt.’,‘Chicago’,‘IL’,‘60601’);
INSERTINTOmembers(lname,fname,address,city,state,zip)VALUES
(‘Bresnahan’,
‘Christine’,‘456OakAve.’,‘Columbus’,‘OH’,‘43201’);
INSERTINTOmembers(lname,fname,address,city,state,zip)VALUES
(‘Bresnahan’,
‘Timothy’,‘456OakAve.’,‘Columbus’,‘OH’,‘43201’);
$
Thescriptworkedexactlyasexpected!Nowyoucaneasilyimportthe members.sqlfile
intoaMySQLdatabasetable(seeChapter25).
Summary
Understanding how the bash shell handles input and output can come in handy when
creating your scripts. You can manipulate both how the script receives data and how it
displaysdata,tocustomizeyourscriptforanyenvironment.Youcanredirecttheinputofa
scriptfromthestandardinput(STDIN)toanyfileonthesystem.Youcanalsoredirectthe
outputofthescriptfromthestandardoutput(STDOUT)toanyfileonthesystem.
Besides the STDOUT, you can redirect any error messages your script generates by
redirecting the STDERR output. This is accomplished by redirecting the file descriptor
associated with the STDERR output, which is file descriptor 2. You can redirect STDERR
outputtothesamefileasthe STDOUToutputortoacompletelyseparatefile.Thisenables
youtoseparatenormalscriptmessagesfromanyerrormessagesgeneratedbythescript.
Thebashshellallowsyoutocreateyourownfiledescriptorsforuseinyourscripts.You
cancreatefiledescriptors3through8andassignthemtoanyoutputfileyoudesire.After
youcreateafiledescriptor,youcanredirecttheoutputofanycommandtoit,usingthe
standardredirectionsymbols.
Thebashshellalsoallowsyoutoredirectinputtoafiledescriptor,providinganeasyway
toreaddatacontainedinafileintoyourscript.Youcanusethe lsofcommandtodisplay
theactivefiledescriptorsinyourshell.
Linuxsystemsprovideaspecialfile,called/dev/null,toallowyoutoredirectoutputthat
youdon’twant.TheLinuxsystemdiscardsanythingredirectedtothe/dev/nullfile.You
canalsousethisfiletoproduceanemptyfilebyredirectingthecontentsofthe/dev/null
filetothefile.
Themktempcommandisahandyfeatureofthebashshellthatallowsyoutoeasilycreate
temporaryfilesanddirectories.Simplyspecifyatemplateforthemktempcommand,andit
createsauniquefileeachtimeyoucallit,basedonthefiletemplateformat.Youcanalso
createtemporaryfilesanddirectoriesinthe/tmpdirectoryontheLinuxsystem,whichisa
speciallocationthatisn’tpreservedbetweensystemboots.
Theteecommandisahandywaytosendoutputbothtothestandardoutputandtoalog
file.Thisenablesyoutodisplaymessagesfromyourscriptonthemonitorandstorethem
inalogfileatthesametime.
In Chapter 16, you’ll see how to control and run your scripts. Linux provides several
differentmethodsforrunningscriptsotherthandirectlyfromthecommandlineinterface
prompt.You’llseehowtoscheduleyourscriptstorunataspecifictime,aswellaslearn
howtopausethemwhilethey’rerunning.
Chapter16
ScriptControl
InThisChapter
1. Handlingsignals
2. Runningscriptsinthebackground
3. Forbiddinghang-ups
4. ControllingaJob
5. Modifyingscriptpriority
6. Automatingscriptexecution
Asyoustartbuildingadvancedscripts,you’llprobablywonderhowtorunand
controlthemonyourLinuxsystem.Sofarinthisbook,theonlywaywe’verun
scriptsisdirectlyfromthecommandlineinterfaceinreal-timemode.Thisisn’tthe
onlywaytorunscriptsinLinux.Quiteafewoptionsareavailableforrunningyour
shellscripts.Therearealsooptionsforcontrollingyourscripts.Variouscontrol
methodsincludesendingsignalstoyourscript,modifyingascript’spriority,and
switchingtherunmodewhileascriptisrunning.Thischapterexaminesthedifferent
waysyoucancontrolyourshellscripts.
HandlingSignals
Linux uses signals to communicate with processes running on the system. Chapter 4
describedthedifferentLinuxsignalsandhowtheLinuxsystemusesthesesignalstostop,
start, and kill processes. You can control the operation of your shell script by
programmingthescripttoperformcertaincommandswhenitreceivesspecificsignals.
Signalingthebashshell
There are more than 30 Linux signals that can be generated by the system and
applications.Table16.1liststhemostcommonLinuxsystemsignalsthatyou’llrunacross
inyourshellscriptwriting.
Table16.1LinuxSignals
Signal
Value
1
SIGHUP
2
SIGINT
3
SIGQUIT
9
SIGKILL
15
SIGTERM
17
SIGSTOP
18
SIGTSTP
19
SIGCONT
Description
Hangsuptheprocess
Interruptstheprocess
Stopstheprocess
Unconditionallyterminatestheprocess
Terminatestheprocessifpossible
Unconditionallystops,butdoesn’tterminate,theprocess
Stopsorpausestheprocess,butdoesn’tterminate
Continuesastoppedprocess
Bydefault,thebashshellignoresany SIGQUIT(3)and SIGTERM(15)signalsitreceives
(soaninteractiveshellcannotbeaccidentallyterminated).However,thebashshelldoes
notignoreanySIGHUP(1)andSIGINT(2)signalsitreceives.
Ifthebashshellreceivesa SIGHUPsignal,suchaswhenyouleaveaninteractiveshell,it
exits.Beforeitexits,however,itpassesthe SIGHUPsignaltoanyprocessesstartedbythe
shell,includinganyrunningshellscripts.
WithaSIGINTsignal,theshellisjustinterrupted.TheLinuxkernelstopsgivingtheshell
processingtimeontheCPU.Whenthishappens,theshellpassestheSIGINTsignaltoany
processesstartedbytheshelltonotifythemofthesituation.
As you probably have noticed, the shell passes these signals on to your shell script
programforprocessing.However,ashellscript’sdefaultbehaviordoesnotgovernthese
signals, which may have an adverse effect on the script’s operation. To avoid this
situation, you can program your script to recognize signals and perform commands to
preparethescriptfortheconsequencesofthesignal.
Generatingsignals
ThebashshellallowsyoutogeneratetwobasicLinuxsignalsusingkeycombinationson
thekeyboard.Thisfeaturecomesinhandyifyouneedtostoporpausearunawayscript.
Interruptingaprocess
The Ctrl+C key combination generates a SIGINT signal and sends it to any processes
currentlyrunningintheshell.Youcantestthisbyrunningacommandthatnormallytakes
alongtimetofinishandpressingtheCtrl+Ckeycombination:
$sleep100
^C
$
The Ctrl+C key combination sends a SIGINT signal, which simply stops the current
process running in the shell. The sleep command pauses the shell’s operation for the
specified number of seconds and returns the shell prompt. By pressing the Ctrl+C key
combinationbeforethetimepassed,thesleepcommandterminatedprematurely.
Pausingaprocess
Instead of terminating a process, you can pause it in the middle of whatever it’s doing.
Sometimes,thiscanbeadangerousthing(forexample,ifascripthasafilelockopenona
crucialsystemfile),butoftenitallowsyoutopeekinsidewhatascriptisdoingwithout
actuallyterminatingtheprocess.
TheCtrl+Zkeycombinationgeneratesa SIGTSTPsignal,stoppinganyprocessesrunning
in the shell. Stopping a process is different than terminating the process. Stopping the
processleavestheprograminmemoryandabletocontinuerunningfromwhereitleftoff.
Inthe“ControllingtheJob”sectionlaterinthischapter,youlearnhowtorestartaprocess
that’sbeenstopped.
WhenyouusetheCtrl+Zkeycombination,theshellinformsyouthattheprocesshasbeen
stopped:
$sleep100
^Z
[1]+Stoppedsleep100
$
Thenumberinthesquarebracketsisthejobnumberassignedbytheshell.Theshellrefers
to each process running in the shell as a job and assigns each job a unique job number
withinthecurrentshell.Itassignsthefirststartedprocessjobnumber1,thesecondjob
number2,andsoon.
Ifyouhaveastoppedjobassignedtoyourshellsession,bashwarnsyouifyoutrytoexit
theshell:
$sleep100
^Z
[1]+Stoppedsleep100
$exit
exit
Therearestoppedjobs.
$
Youcanviewthestoppedjobsusingthepscommand:
$sleep100
^Z
[1]+Stoppedsleep100
$
$ps-l
FSUIDPIDPPIDCPRINIADDRSZWCHANTTYTIMECMD
0S501243124300800-27118waitpts/000:00:00bash
0T501245624310800-25227signalpts/000:00:00sleep
0R501245824310800-27034-pts/000:00:00ps
$
Inthe Scolumn(processstate),the pscommandshowsthestoppedjob’sstateasT.This
indicatesthecommandiseitherbeingtracedorisstopped.
If you really want to exit the shell with a stopped job still active, just type the exit
command again. The shell exits, terminating the stopped job. Alternately, now that you
knowthePIDofthestoppedjob,youcanusethekillcommandtosendaSIGKILLsignal
toterminateit:
$kill-92456
$
[1]+Killedsleep100
$
Whenyoukillthejob,initiallyyoudon’tgetanyresponse.However,thenexttimeyoudo
something that produces a shell prompt (such as pressing the Enter key), you’ll see a
messageindicatingthatthejobwaskilled.Eachtimetheshellproducesaprompt,italso
displaysthestatusofanyjobsthathavechangedstatesintheshell.Afteryoukillajob,
thenexttimeyouforcetheshelltoproduceaprompt,itdisplaysamessageshowingthat
thejobwaskilledwhilerunning.
Trappingsignals
Insteadofallowingyourscripttoleavesignalsungoverned,youcantrapthemwhenthey
appear and perform other commands. The trap command allows you to specify which
Linux signals your shell script can watch for and intercept from the shell. If the script
receivesasignallistedinthe trapcommand,itpreventsitfrombeingprocessedbythe
shellandinsteadhandlesitlocally.
Theformatofthetrapcommandis:
trapcommandssignals
On the trap command line, you just list the commands you want the shell to execute,
alongwithaspace-separatedlistofsignalsyouwanttotrap.Youcanspecifythesignals
eitherbytheirnumericvalueorbytheirLinuxsignalname.
Here’s a simple example of using the trap command to capture the SIGINT signal and
governthescript’sbehaviorwhenthesignalissent:
$cattest1.sh
#!/bin/bash
#Testingsignaltrapping
#
trap“echo‘Sorry!IhavetrappedCtrl-C”’SIGINT
#
echoThisisatestscript
#
count=1
while[$count-le10]
do
echo“Loop#$count”
sleep1
count=$[$count+1]
done
#
echo“Thisistheendofthetestscript”
#
The trap command used in this example displays a simple text message each time it
detects the SIGINT signal. Trapping this signal makes this script impervious to the user
attemptingtostoptheprogrambyusingthebashshellkeyboardCtrl+Ccommand:
$./test1.sh
Thisisatestscript
Loop#1
Loop#2
Loop#3
Loop#4
Loop#5
^CSorry!IhavetrappedCtrl-C
Loop#6
Loop#7
Loop#8
^CSorry!IhavetrappedCtrl-C
Loop#9
Loop#10
Thisistheendofthetestscript
$
EachtimetheCtrl+Ckeycombinationwasused,thescriptexecutedthe echostatement
specifiedinthe trapcommandinsteadofnotmanagingthesignalandallowingtheshell
tostopthescript.
Trappingascriptexit
Besidestrappingsignalsinyourshellscript,youcantrapthemwhentheshellscriptexits.
Thisisaconvenientwaytoperformcommandsjustastheshellfinishesitsjob.
Totraptheshellscriptexiting,justaddtheEXITsignaltothetrapcommand:
$cattest2.sh
#!/bin/bash
#Trappingthescriptexit
#
trap“echoGoodbye…”EXIT
#
count=1
while[$count-le5]
do
echo“Loop#$count”
sleep1
count=$[$count+1]
done
#
$
$./test2.sh
Loop#1
Loop#2
Loop#3
Loop#4
Loop#5
Goodbye…
$
Whenthescriptgetstothenormalexitpoint,thetrapistriggered,andtheshellexecutes
the command you specify on the trap command line. The EXIT trap also works if you
prematurelyexitthescript:
$./test2.sh
Loop#1
Loop#2
Loop#3
^CGoodbye…
$
Because the SIGINT signal isn’t listed in the trap command list, when the Ctrl+C key
combinationisusedtosendthatsignal,thescriptexits.However,beforethescriptexits,
becausetheEXITistrapped,theshellexecutesthetrapcommand.
Modifyingorremovingatrap
Tohandletrapsdifferentlyinvarioussectionsofyourshellscript,yousimplyreissuethe
trapcommandwithnewoptions:
$cattest3.sh
#!/bin/bash
#Modifyingasettrap
#
trap“echo‘Sorry…Ctrl-Cistrapped.”’SIGINT
#
count=1
while[$count-le5]
do
echo“Loop#$count”
sleep1
count=$[$count+1]
done
#
trap“echo‘Imodifiedthetrap!”’SIGINT
#
count=1
while[$count-le5]
do
echo“SecondLoop#$count”
sleep1
count=$[$count+1]
done
#
$
After the signal trap is modified, the script manages the signal or signals differently.
However,ifasignalisreceivedbeforethetrapismodified,thescriptprocessesitperthe
originaltrapcommand:
$./test3.sh
Loop#1
Loop#2
Loop#3
^CSorry…Ctrl-Cistrapped.
Loop#4
Loop#5
SecondLoop#1
SecondLoop#2
^CImodifiedthetrap!
SecondLoop#3
SecondLoop#4
SecondLoop#5
$
Youcanalsoremoveasettrap.Simplyaddtwodashesafterthe trapcommandandalist
ofthesignalsyouwanttoreturntodefaultbehavior:
$cattest3b.sh
#!/bin/bash
#Removingasettrap
#
trap“echo‘Sorry…Ctrl-Cistrapped.”’SIGINT
#
count=1
while[$count-le5]
do
echo“Loop#$count”
sleep1
count=$[$count+1]
done
#
#Removethetrap
trap—SIGINT
echo“Ijustremovedthetrap”
#
count=1
while[$count-le5]
do
echo“SecondLoop#$count”
sleep1
count=$[$count+1]
done
#
$./test3b.sh
Loop#1
Loop#2
Loop#3
Loop#4
Loop#5
Ijustremovedthetrap
SecondLoop#1
SecondLoop#2
SecondLoop#3
^C
$
Tip
Youcanuseasingledashinsteadofadoubledashafterthetrapcommandtoreturn
signalstotheirdefaultbehavior.Boththesingleanddoubledashworkproperly.
Afterthesignaltrapisremoved,thescripthandlestheSIGINTsignalinitsdefaultmanner,
terminating the script. However, if a signal is received before the trap is removed, the
scriptprocessesitpertheoriginaltrapcommand:
$./test3b.sh
Loop#1
Loop#2
Loop#3
^CSorry…Ctrl-Cistrapped.
Loop#4
Loop#5
Ijustremovedthetrap
SecondLoop#1
SecondLoop#2
^C
$
In this example, the first Ctrl+C key combination was used to attempt to terminate the
script prematurely. Because the signal was received before the trap was removed, the
script executed the command specified in the trap. After the script executed the trap
removal,thenCtrl+Ccouldprematurelyterminatethescript.
RunningScriptsinBackgroundMode
Sometimes, running a shell script directly from the command line interface is
inconvenient.Somescriptscantakealongtimetoprocess,andyoumaynotwanttotieup
thecommandlineinterfacewaiting.Whilethescriptisrunning,youcan’tdoanythingelse
inyourterminalsession.Fortunately,there’sasimplesolutiontothatproblem.
Whenyouusethe pscommand,youseeawholebunchofdifferentprocessesrunningon
theLinuxsystem.Obviously,alltheseprocessesarenotrunningonyourterminalmonitor.
Thisiscalledrunningprocessesinthebackground.Inbackgroundmode,aprocessruns
without being associated with a STDIN, STDOUT, and STDERR on a terminal session (see
Chapter15).
Youcanexploitthisfeaturewithyourshellscriptsaswell,allowingthemtorunbehind
thescenesandnotlockupyourterminalsession.Thefollowingsectionsdescribehowto
runyourscriptsinbackgroundmodeonyourLinuxsystem.
Runninginthebackground
Runningashellscriptinbackgroundmodeisafairlyeasythingtodo.Torunashellscript
in background mode from the command line interface, just place an ampersand symbol
(&)afterthecommand:
$cattest4.sh
#!/bin/bash
#Testrunninginthebackground
#
count=1
while[$count-le10]
do
sleep1
count=$[$count+1]
done
#
$
$./test4.sh&
[1]3231
$
Whenyouplacetheampersandsymbolafteracommand,itseparatesthecommandfrom
thebashshellandrunsitasaseparatebackgroundprocessonthesystem.Thefirstthing
thatdisplaysistheline:
[1]3231
The number in the square brackets is the job number assigned by the shell to the
backgroundprocess.ThenextnumberistheProcessID(PID)theLinuxsystemassignsto
theprocess.EveryprocessrunningontheLinuxsystemmusthaveauniquePID.
Assoonasthesystemdisplaystheseitems,anewcommandlineinterfacepromptappears.
Youarereturnedtotheshell,andthecommandyouexecutedrunssafelyinbackground
mode.Atthispoint,youcanenternewcommandsattheprompt.
Whenthebackgroundprocessfinishes,itdisplaysamessageontheterminal:
[1]Done./test4.sh
Thisshowsthejobnumberandthestatusofthejob(Done),alongwiththecommandused
tostartthejob.
Beawarethatwhilethebackgroundprocessisrunning,itstillusesyourterminalmonitor
forSTDOUTandSTDERRmessages:
$cattest5.sh
#!/bin/bash
#Testrunninginthebackgroundwithoutput
#
echo“Startthetestscript”
count=1
while[$count-le5]
do
echo“Loop#$count”
sleep5
count=$[$count+1]
done
#
echo“Testscriptiscomplete”
#
$
$./test5.sh&
[1]3275
$Startthetestscript
Loop#1
Loop#2
Loop#3
Loop#4
Loop#5
Testscriptiscomplete
[1]Done./test5.sh
$
You’ll notice from the example that the output from the test5.sh script displays. The
outputintermixeswiththeshellprompt,whichiswhy Start the test scriptappears
nexttothe$prompt.
Youcanstillissuecommandswhilethisoutputisoccurring:
$./test5.sh&
[1]3319
$Startthetestscript
Loop#1
Loop#2
Loop#3
lsmyprog*
myprogmyprog.c
$Loop#4
Loop#5
Testscriptiscomplete
[1]+Done./test5.sh
$$
Whilethe test5.sh script is running in the background, the command ls myprog*was
entered.Thescript’soutput,thetypedcommand,andthecommand’soutputallintermixed
with each other’s output display. This can be confusing! It is a good idea to redirect
STDOUT and STDERR for scripts you will be running in the background (Chapter 15) to
avoidthismessyoutput.
Runningmultiplebackgroundjobs
You can start any number of background jobs at the same time from the command line
prompt:
$./test6.sh&
[1]3568
$ThisisTestScript#1
$./test7.sh&
[2]3570
$ThisisTestScript#2
$./test8.sh&
[3]3573
$And…anotherTestscript
$./test9.sh&
[4]3576
$Then…therewasonemoretestscript
$
Eachtimeyoustartanewjob,theLinuxsystemassignsitanewjobnumberandPID.You
canseethatallthescriptsarerunningusingthepscommand:
$ps
PIDTTYTIMECMD
2431pts/000:00:00bash
3568pts/000:00:00test6.sh
3570pts/000:00:00test7.sh
3573pts/000:00:00test8.sh
3574pts/000:00:00sleep
3575pts/000:00:00sleep
3576pts/000:00:00test9.sh
3577pts/000:00:00sleep
3578pts/000:00:00sleep
3579pts/000:00:00ps
$
Youmustbecarefulwhenusingbackgroundprocessesfromaterminalsession.Noticein
the output from the ps command that each of the background processes is tied to the
terminalsession(pts/0)terminal.Iftheterminalsessionexits,thebackgroundprocessalso
exits.
Note
Earlierinthischapterwementionedthatwhenyouattempttoexitaterminalsession,
awarningisissuediftherearestoppedprocesses.However,withbackground
processes,onlysometerminalemulatorsremindyouthatabackgroundjobis
running,beforeyouattempttoexittheterminalsession.
Ifyouwantyourscripttocontinuerunninginbackgroundmodeafteryouhaveloggedoff
the console, there’s something else you need to do. The next section discusses that
process.
RunningScriptswithoutaHang-Up
Sometimes,youmaywanttostartashellscriptfromaterminalsessionandletthescript
runinbackgroundmodeuntilitfinishes,evenifyouexittheterminalsession.Youcando
thisbyusingthenohupcommand.
The nohupcommandrunsanothercommandblockingany SIGHUPsignalsthataresentto
theprocess.Thispreventstheprocessfromexitingwhenyouexityourterminalsession.
Theformatusedforthenohupcommandisasfollows:
$nohup./test1.sh&
[1]3856
$nohup:ignoringinputandappendingoutputto‘nohup.out’
$
Aswithanormalbackgroundprocess,theshellassignsthecommandajobnumber,and
the Linux system assigns a PID number. The difference is that when you use the nohup
command,thescriptignoresany SIGHUPsignalssentbytheterminalsessionifyouclose
thesession.
Becausethenohupcommanddisassociatestheprocessfromtheterminal,theprocessloses
the STDOUT and STDERR output links. To accommodate any output generated by the
command,the nohupcommandautomaticallyredirects STDOUTand STDERRmessagestoa
file,callednohup.out.
Note
Ifyourunanothercommandusingnohup,theoutputisappendedtotheexisting
nohup.outfile.Becarefulwhenrunningmultiplecommandsfromthesame
directory,becausealltheoutputissenttothesamenohup.outfile,whichcanget
confusing.
The nohup.out file contains all the output that would normally be sent to the terminal
monitor.Aftertheprocessfinishesrunning,youcanviewthenohup.outfilefortheoutput
results:
$catnohup.out
Thisisatestscript
Loop1
Loop2
Loop3
Loop4
Loop5
Loop6
Loop7
Loop8
Loop9
Loop10
Thisistheendofthetestscript
$
Theoutputappearsinthenohup.outfilejustasiftheprocessranonthecommandline.
ControllingtheJob
Earlier in this chapter, you saw how to use the Ctrl+C key combination to stop a job
runningintheshell.Afteryoustopajob,theLinuxsystemletsyoueitherkillorrestartit.
You can kill the process by using the kill command. Restarting a stopped process
requiresthatyousenditaSIGCONTsignal.
The function of starting, stopping, killing, and resuming jobs is called jobcontrol.With
jobcontrol,youhavefullcontroloverhowprocessesruninyourshellenvironment.This
sectiondescribesthecommandsusedtoviewandcontroljobsrunninginyourshell.
Viewingjobs
Thekeycommandforjobcontrolisthejobscommand.Thejobscommandallowsyouto
viewthecurrentjobsbeinghandledbytheshell:
$cattest10.sh
#!/bin/bash
#Testjobcontrol
#
echo“ScriptProcessID:$$”
#
count=1
while[$count-le10]
do
echo“Loop#$count”
sleep10
count=$[$count+1]
done
#
echo“Endofscript…”
#
$
The script uses the $$ variable to display the PID that the Linux system assigns to the
script;thenitgoesintoaloop,sleepingfor10secondsatatimeforeachiteration.
YoucanstartthescriptfromthecommandlineinterfaceandthenstopitusingtheCtrl+Z
keycombination:
$./test10.sh
ScriptProcessID:1897
Loop#1
Loop#2
^Z
[1]+Stopped./test10.sh
$
Usingthesamescript,anotherjobisstartedasabackgroundprocess,usingtheampersand
symbol. To make life a little easier, the output of that script is redirected to a file so it
doesn’tappearonthescreen:
$./test10.sh>test10.out&
[2]1917
$
Thejobscommandenablesyoutoviewthejobsassignedtotheshell.Thejobscommand
shows both the stopped and the running jobs, along with their job numbers and the
commandsusedinthejobs:
$jobs
[1]+Stopped./test10.sh
[2]-Running./test10.sh>test10.out&
$
Youcanviewthevariousjobs’PIDsbyaddingthe-lparameter(lowercaseL)onthejobs
command:
$jobs-l
[1]+1897Stopped./test10.sh
[2]-1917Running./test10.sh>test10.out&
$
The jobs command uses a few different command line parameters, as shown in Table
16.2.
Table16.2ThejobsCommandParameters
Parameter
-l
-n
-p
-r
-s
Description
ListsthePIDoftheprocessalongwiththejobnumber
Listsonlyjobsthathavechangedtheirstatussincethelastnotificationfrom
theshell
ListsonlythePIDsofthejobs
Listsonlytherunningjobs
Listsonlystoppedjobs
Youprobablynoticedtheplusandminussignsinthejobscommandoutput.Thejobwith
the plus sign is considered the default job. It would be the job referenced by any job
controlcommandsifajobnumberwasn’tspecifiedinthecommandline.
Thejobwiththeminussignisthejobthatwouldbecomethedefaultjobwhenthecurrent
defaultjobfinishesprocessing.Therewillbeonlyonejobwiththeplussignandonejob
withtheminussignatanytime,nomatterhowmanyjobsarerunningintheshell.
The following is an example showing how the next job in line takes over the default
status, when the default job is removed. Three separate processes are started in the
background. The jobs command listing shows the three processes, their PID, and their
status.Notethatthedefaultprocess(theonelistedwiththeplussign)isthelastprocess
started,job#3.
$./test10.sh>test10a.out&
[1]1950
$./test10.sh>test10b.out&
[2]1952
$./test10.sh>test10c.out&
[3]1955
$
$jobs-l
[1]1950Running./test10.sh>test10a.out&
[2]-1952Running./test10.sh>test10b.out&
[3]+1955Running./test10.sh>test10c.out&
$
Usingthe killcommandtosenda SIGHUPsignaltothedefaultprocesscausesthejobto
terminate.Inthenext jobslisting,thejobthatpreviouslyhadtheminussignnowhasthe
plussignandisthedefaultjob:
$kill1955
$
[3]+Terminated./test10.sh>test10c.out
$
$jobs-l
[1]-1950Running./test10.sh>test10a.out&
[2]+1952Running./test10.sh>test10b.out&
$
$kill1952
$
[2]+Terminated./test10.sh>test10b.out
$
$jobs-l
[1]+1950Running./test10.sh>test10a.out&
$
Althoughchangingabackgroundjobtothedefaultprocessisinteresting,itdoesn’tseem
very useful. In the next section, you learn how to use commands to interact with the
defaultprocessusingnoPIDorjobnumber.
Restartingstoppedjobs
Underbashjobcontrol,youcanrestartanystoppedjobaseitherabackgroundprocessor
a foreground process. A foreground process takes over control of the terminal you’re
workingon,sobecarefulaboutusingthatfeature.
Torestartajobinbackgroundmode,usethebgcommand:
$./test11.sh
^Z
[1]+Stopped./test11.sh
$
$bg
[1]+./test11.sh&
$
$jobs
[1]+Running./test11.sh&
$
Becausethejobwasthedefaultjob,indicatedbytheplussign,onlythe bgcommandwas
needed to restart it in background mode. Notice that no PID is listed when the job is
movedintobackgroundmode.
Ifyouhaveadditionaljobs,youneedtousethejobnumberalongwiththebgcommand:
$./test11.sh
^Z
[1]+Stopped./test11.sh
$
$./test12.sh
^Z
[2]+Stopped./test12.sh
$
$bg2
[2]+./test12.sh&
$
$jobs
[1]+Stopped./test11.sh
[2]-Running./test12.sh&
$
Thecommand bg2wasusedtosendthesecondjobintobackgroundmode.Noticethat
whenthe jobscommandwasused,itlistedbothjobswiththeirstatus,eventhoughthe
defaultjobisnotcurrentlyinbackgroundmode.
Torestartajobinforegroundmode,usethefgcommand,alongwiththejobnumber:
$fg2
./test12.sh
Thisisthescript’send…
$
Becausethejobisrunninginforegroundmode,thecommandlineinterfacepromptdoes
notappearuntilthejobfinishes.
BeingNice
In a multitasking operating system (which Linux is), the kernel is responsible for
assigningCPUtimeforeachprocessrunningonthesystem.Theschedulingpriorityisthe
amountofCPUtimethekernelassignstotheprocessrelativetotheotherprocesses.By
default,allprocessesstartedfromtheshellhavethesameschedulingpriorityontheLinux
system.
The scheduling priority is an integer value, from -20 (the highest priority) to +19 (the
lowestpriority).Bydefault,thebashshellstartsallprocesseswithaschedulingpriorityof
0.
Tip
It’sconfusingtorememberthat-20,thelowestvalue,isthehighestpriorityand19,
thehighestvalue,isthelowestpriority.Justrememberthephrase,“Niceguysfinish
last.”The“nicer”orhigheryouareinvalue,theloweryourchanceofgettingthe
CPU.
Sometimes,youwanttochangethepriorityofashellscript,eitherloweringitspriorityso
itdoesn’ttakeasmuchprocessingpowerawayfromotherprocessesorgivingitahigher
prioritysoitgetsmoreprocessingtime.Youcandothisbyusingthenicecommand.
Usingthenicecommand
Thenicecommandallowsyoutosettheschedulingpriorityofacommandasyoustartit.
Tomakeacommandrunwithlesspriority,justusethe -ncommandlineoptionfor nice
tospecifyanewprioritylevel:
$nice-n10./test4.sh>test4.out&
[1]4973
$
$ps-p4973-opid,ppid,ni,cmd
PIDPPIDNICMD
4973472110/bin/bash./test4.sh
$
Notice that you must use the nice command on the same line as the command you are
starting.Theoutputfromthe pscommandconfirmsthatthenicevalue(column NI)has
beensetto10.
The nice command causes the script to run at a lower priority. However, if you try to
increasethepriorityofoneofyourcommands,youmightbeinforasurprise:
$nice-n-10./test4.sh>test4.out&
[1]4985
$nice:cannotsetniceness:Permissiondenied
[1]+Donenice-n-10./test4.sh>test4.out
$
The nice command prevents normal system users from increasing the priority of their
commands.Noticethatthejobdoesrun,eventhoughtheattempttoraiseitsprioritywith
thenicecommandfailed.
You don’t have to use the -n option with the nice command. You can simply type the
priorityprecededbyadash:
$nice-10./test4.sh>test4.out&
[1]4993
$
$ps-p4993-opid,ppid,ni,cmd
PIDPPIDNICMD
4993472110/bin/bash./test4.sh
$
However,thiscangetconfusingwhenthepriorityisanegativenumber,becauseyoumust
haveadouble-dash.It’sbestjusttousethe-noptiontoavoidconfusion.
Usingtherenicecommand
Sometimes,you’dliketochangethepriorityofacommandthat’salreadyrunningonthe
system. That’s what the renice command is for. It allows you to specify the PID of a
runningprocesstochangeitspriority:
$./test11.sh&
[1]5055
$
$ps-p5055-opid,ppid,ni,cmd
PIDPPIDNICMD
505547210/bin/bash./test11.sh
$
$renice-n10-p5055
5055:oldpriority0,newpriority10
$
$ps-p5055-opid,ppid,ni,cmd
PIDPPIDNICMD
5055472110/bin/bash./test11.sh
$
The renice command automatically updates the scheduling priority of the running
process.Aswiththenicecommand,therenicecommandhassomelimitations:
Youcanonlyreniceprocessesthatyouown.
Youcanonlyreniceyourprocessestoalowerpriority.
Therootusercanreniceanyprocesstoanypriority.
Ifyouwanttofullycontrolrunningprocesses,youmustbeloggedinastherootaccount
orusethesudocommand.
RunningLikeClockwork
Whenyoustartworkingwithscripts,youmaywanttorunascriptatapresettime,usually
at a time when you’re not there. The Linux system provides a couple of ways to run a
script at a preselected time: the at command and the cron table. Each method uses a
different technique for scheduling when and how often to run scripts. The following
sectionsdescribeeachofthesemethods.
Schedulingajobusingtheatcommand
The atcommandallowsyoutospecifyatimewhentheLinuxsystemwillrunascript.
The atcommandsubmitsajobtoaqueuewithdirectionsonwhentheshellshouldrun
thejob.The atdaemon,atd,runsinthebackgroundandchecksthejobqueueforjobsto
run.MostLinuxdistributionsstartthisdaemonautomaticallyatboottime.
The atd daemon checks a special directory on the system (usually /var/spool/at) for
jobssubmittedusingthe atcommand.Bydefault,the atddaemonchecksthisdirectory
every60seconds.Whenajobispresent,the atddaemonchecksthetimethejobissetto
berun.Ifthetimematchesthecurrenttime,theatddaemonrunsthejob.
The following sections describe how to use the at command to submit jobs to run and
howtomanagethesejobs.
Understandingtheatcommandformat
Thebasicatcommandformatisprettysimple:
at[-ffilename]time
By default, the at command submits input from STDIN to the queue. You can specify a
filenameusedtoreadcommands(yourscriptfile)usingthe-fparameter.
The time parameter specifies when you want the Linux system to run the job. If you
specifyatimethathasalreadypassed,the atcommandrunsthejobatthattimeonthe
nextday.
Youcangetprettycreativewithhowyouspecifythetime.The atcommandrecognizes
lotsofdifferenttimeformats:
Astandardhourandminute,suchas10:15
AnAM/PMindicator,suchas10:15PM
Aspecificnamedtime,suchasnow,noon,midnight,orteatime(4PM)
Inadditiontospecifyingthetimetorunthejob,youcanalsoincludeaspecificdate,using
afewdifferentdateformats:
Astandarddateformat,suchasMMDDYY,MM/DD/YY,orDD.MM.YY
Atextdate,suchasJul4orDec25,withorwithouttheyear
Atimeincrement:
Now+25minutes
10:15PMtomorrow
10:15+7days
Whenyouusetheatcommand,thejobissubmittedintoajobqueue.Thejobqueueholds
thejobssubmittedbythe atcommandforprocessing.Thereare26differentjobqueues
availablefordifferentprioritylevels.Jobqueuesarereferencedusinglowercaseletters,a
throughz,anduppercaselettersAthroughZ.
Note
Afewyearsago,thebatchcommandwasanothermethodthatallowedascripttobe
runatalatertime.Thebatchcommandwasuniquebecauseyoucouldschedulea
scripttorunwhenthesystemwasatalowerusagelevel.However,nowadays,the
batchcommandisjustsimplyascript,/usr/bin/batch,thatcallstheatcommand
andsubmitsyourjobtothebqueue.
Thehigheralphabeticallythejobqueue,thelowerthepriority(higher nicevalue)thejob
willrununder.Bydefault, atjobsaresubmittedtothe atjobaqueue.Ifyouwanttorun
ajobatalowerpriority,youcanspecifyadifferentqueueletterusingthe-qparameter.
Retrievingjoboutput
When the job runs on the Linux system, there’s no monitor associated with the job.
Instead, the Linux system uses the e-mail address of the user who submitted the job as
STDOUTandSTDERR.AnyoutputdestinedtoSTDOUTor STDERRismailedtotheuserviathe
mailsystem.
Here’s a simple example using the at command to schedule a job to run on a CentOS
distribution:
$cattest13.sh
#!/bin/bash
#Testusingatcommand
#
echo“Thisscriptranat$(date+%B%d,%T)”
echo
sleep5
echo“Thisisthescript’send…”
#
$at-ftest13.shnow
job7at2015-07-1412:38
$
The atcommanddisplaysthejobnumberassignedtothejobalongwiththetimethejob
isscheduledtorun.The-foptiontellswhatscriptfiletouseandthenowtimedesignation
directsattorunthescriptimmediately.
Usinge-mailfortheatcommand’soutputisinconvenientatbest.Theatcommandsends
e-mailviathesendmailapplication.Ifyoursystemdoesnotusesendmail,youwon’tget
anyoutput!Therefore,it’sbesttoredirectSTDOUTandSTDERRinyourscripts(seeChapter
15)whenusingtheatcommand,asthefollowingexampleshows:
$cattest13b.sh
#!/bin/bash
#Testusingatcommand
#
echo“Thisscriptranat$(date+%B%d,%T)”>test13b.out
echo>>test13b.out
sleep5
echo“Thisisthescript’send…”>>test13b.out
#
$
$at-M-ftest13b.shnow
job8at2015-07-1412:48
$
$cattest13b.out
ThisscriptranatJuly14,12:48:18
Thisisthescript’send…
$
If you don’t want to use e-mail or redirection with at, it is best to add the -Moptionto
suppressanyoutputgeneratedbyjobsusingtheatcommand.
Listingpendingjobs
Theatqcommandallowsyoutoviewwhatjobsarependingonthesystem:
$at-M-ftest13b.shteatime
job17at2015-07-1416:00
$
$at-M-ftest13b.shtomorrow
job18at2015-07-1513:03
$
$at-M-ftest13b.sh13:30
job19at2015-07-1413:30
$
$at-M-ftest13b.shnow
job20at2015-07-1413:03
$
$atq
202015-07-1413:03=Christine
182015-07-1513:03aChristine
172015-07-1416:00aChristine
192015-07-1413:30aChristine
$
Thejoblistingshowsthejobnumber,thedateandtimethesystemwillrunthejob,and
thejobqueuethejobisstoredin.
Removingjobs
Afteryouknowtheinformationaboutwhatjobsarependinginthejobqueues,youcan
usetheatrmcommandtoremoveapendingjob:
$atq
182015-07-1513:03aChristine
172015-07-1416:00aChristine
192015-07-1413:30aChristine
$
$atrm18
$
$atq
172015-07-1416:00aChristine
192015-07-1413:30aChristine
$
Just specify the job number you want to remove. You can only remove jobs that you
submitforexecution.Youcan’tremovejobssubmittedbyothers.
Schedulingregularscripts
Usingtheatcommandtoscheduleascripttorunatapresettimeisgreat,butwhatifyou
needthatscripttorunatthesametimeeverydayoronceaweekoronceamonth?Instead
ofhavingtocontinuallysubmitatjobs,youcanuseanotherfeatureoftheLinuxsystem.
TheLinuxsystemusesthecronprogramtoallowyoutoschedulejobsthatneedtorunon
aregularbasis.Thecronprogramrunsinthebackgroundandchecksspecialtables,called
crontables,forjobsthatarescheduledtorun.
Lookingatthecrontable
Thecrontableusesaspecialformatforallowingyoutospecifywhenajobshouldberun.
Theformatforthecrontableis:
minhourdayofmonthmonthdayofweekcommand
The crontableallowsyoutospecifyentriesasspecificvalues,rangesofvalues(suchas
1–5),orasawildcardcharacter(theasterisk).Forexample,ifyouwanttorunacommand
at10:15oneveryday,youwouldusethiscrontableentry:
1510***command
Thewildcardcharacterusedinthedayofmonth,month,anddayofweekfieldsindicatesthat
cronwillexecutethecommandeverydayofeverymonthat10:15.Tospecifyacommand
torunat4:15PMeveryMonday,youwouldusethefollowing:
1516**1command
Youcanspecifythedayofweekentryaseitherathree-charactertextvalue(mon,tue,wed,
thu,fri,sat,sun)orasanumericvalue,with0beingSundayand6beingSaturday.
Here’sanotherexample:toexecuteacommandat12noononthefirstdayofeverymonth,
youwouldusethefollowingformat:
00121**command
Thedayofmonthentryspecifiesadatevalue(1–31)forthemonth.
Note
Theastutereadermightbewonderingjusthowyouwouldbeabletosetacommand
toexecuteonthelastdayofeverymonthbecauseyoucan’tsetthedayofmonthvalue
tocovereverymonth.ThisproblemhasplaguedLinuxandUnixprogrammers,and
hasspawnedquiteafewdifferentsolutions.Acommonmethodistoaddanif-then
statementthatusesthedatecommandtocheckiftomorrow’sdateis01:
1. 0012***if[”date+%d-dtomorrow”=01];then;command
Thischeckseverydayat12noontoseeifit’sthelastdayofthemonth,andifso,
cronrunsthecommand.
Thecommandlistmustspecifythefullcommandpathnameorshellscripttorun.Youcan
addanycommandlineparametersorredirectionsymbolsyoulike,asaregularcommand
line:
1510***/home/rich/test4.sh>test4out
Thecronprogramrunsthescriptusingtheuseraccountthatsubmittedthejob.Thus,you
musthavetheproperpermissionstoaccessthecommandandoutputfilesspecifiedinthe
commandlisting.
Buildingthecrontable
Each system user can have their own cron table (including the root user) for running
scheduledjobs.Linuxprovidesthe crontabcommandforhandlingthe crontable.Tolist
anexistingcrontable,usethe-lparameter:
$crontab-l
nocrontabforrich
$
Bydefault,eachuser’scrontablefiledoesn’texist.Toaddentriestoyour crontable,use
the -e parameter. When you do that, the crontab command starts a text editor (see
Chapter10)withtheexistingcrontable(oranemptyfileifitdoesn’tyetexist).
Viewingcrondirectories
Whenyoucreateascriptthathaslesspreciseexecutiontimeneeds,itiseasiertouseone
ofthepre-configuredcronscriptdirectories.Therearefourbasicdirectories:hourly,daily,
monthly,andweekly.
$ls/etc/cron.*ly
/etc/cron.daily:
cupsmakewhatis.cronprelinktmpwatch
logrotatemlocate.cronreadahead.cron
/etc/cron.hourly:
0anacron
/etc/cron.monthly:
readahead-monthly.cron
/etc/cron.weekly:
$
Thus,ifyouhaveascriptthatneedstoberunonetimeperday,justcopythescripttothe
dailydirectoryandcronexecutesiteachday.
Lookingattheanacronprogram
The only problem with the cron program is that it assumes that your Linux system is
operational 24 hours a day, 7 days a week. Unless you’re running Linux in a server
environment,thismaynotnecessarilybetrue.
IftheLinuxsystemisturnedoffatthetimeajobisscheduledtoruninthecrontable,the
jobdoesn’trun.The cronprogramdoesn’tretroactivelyrunmissedjobswhenthesystem
isturnedbackon.Toresolvethisissue,manyLinuxdistributionsalsoincludetheanacron
program.
Ifanacrondeterminesthatajobhasmissedascheduledrunning,itrunsthejobassoonas
possible.ThismeansthatifyourLinuxsystemisturnedoffforafewdays,whenitstarts
backup,anyjobsscheduledtorunduringthetimeitwasoffareautomaticallyrun.
Thisfeatureisoftenusedforscriptsthatperformroutinelogmaintenance.Ifthesystemis
always off when the script should run, the log files would never get trimmed and could
growtoundesirablesizes.With anacron,you’reguaranteedthatthelogfilesaretrimmed
atleasteachtimethesystemisstarted.
The anacron program deals only with programs located in the cron directories, such as
/etc/cron.monthly. It uses timestamps to determine if the jobs have been run at the
properscheduledinterval.Atimestampfileexistsforeachcrondirectoryandislocatedin
/var/spool/anacron:
$sudocat/var/spool/anacron/cron.monthly
20150626
$
Theanacronprogramhasitsowntable(usuallylocatedat/etc/anacrontab)tocheckthe
jobdirectories:
$sudocat/etc/anacrontab
#/etc/anacrontab:configurationfileforanacron
#Seeanacron(8)andanacrontab(5)fordetails.
SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
#themaximalrandomdelayaddedtothebasedelayofthejobs
RANDOM_DELAY=45
#thejobswillbestartedduringthefollowinghoursonly
START_HOURS_RANGE=3-22
#periodindaysdelayinminutesjob-identifiercommand
15cron.dailynicerun-parts/etc/cron.daily
725cron.weeklynicerun-parts/etc/cron.weekly
@monthly45cron.monthlynicerun-parts/etc/cron.monthly
$
Thebasicformatoftheanacrontableisslightlydifferentfromthatofthecrontable:
perioddelayidentifiercommand
Theperiodentrydefineshowoftenthejobsshouldberun,specifiedindays.Theanacron
programusesthisentrytocheckagainstthejobs’timestampfile.Thedelayentryspecifies
howmanyminutesafterthesystemstartstheanacronprogramshouldrunmissedscripts.
The command entry contains the run-parts program and a cron script directory name.
Therun-partsprogramisresponsibleforrunninganyscriptinthedirectorypassedtoit.
Noticethatanacrondoesnotrunthescriptslocatedin/etc/cron.hourly.Thisisbecause
theanacronprogramdoesnotdealwithscriptsthathaveexecutiontimeneedsoflessthan
daily.
Theidentifierentryisauniquenon-blankcharacterstring—forexample,cron-weekly.It
isusedtouniquelyidentifythejobinlogmessagesanderrore-mails.
Startingscriptswithanewshell
The ability to run a script every time a user starts a new bash shell (even just when a
specific user starts a bash shell) can come in handy. Sometimes, you want to set shell
featuresforashellsessionorjustensurethataspecificfilehasbeenset.
Recall the startup files run when a user logs into the bash shell (covered in detail in
Chapter6).Also,rememberthatnoteverydistributionhasallthestartupfiles.Essentially,
thefirstfilefoundinthefollowingorderedlistisrunandtherestareignored:
$HOME/.bash_profile
$HOME/.bash_login
$HOME/.profile
Therefore,youshouldplaceanyscriptsyouwantrunatlogintimeinthefirstfilelisted.
Thebashshellrunsthe .bashrcfileanytimeanewshellisstarted.Youcantestthisby
addingasimpleechostatementtothe .bashrcfileinyourhomedirectoryandstartinga
newshell:
$cat.bashrc
#.bashrc
#Sourceglobaldefinitions
if[-f/etc/bashrc];then
./etc/bashrc
fi
#Userspecificaliasesandfunctions
echo“I’minanewshell!”
$
$bash
I’minanewshell!
$
$exit
exit
$
The .bashrc file is also typically run from one of the bash startup files. Because the
.bashrcfilerunsbothwhenyoulogintothebashshellandwhenyoustartabashshell,if
youneedascripttoruninbothinstances,placeyourshellscriptinsidethisfile.
Summary
TheLinuxsystemallowsyoutocontrolyourshellscriptsbyusingsignals.Thebashshell
acceptssignalsandpassesthemontoanyprocessrunningundertheshellprocess.Linux
signals allow you to easily kill a runaway process or temporarily pause a long-running
process.
Youcanusethe trapstatementinyourscriptstocatchsignalsandperformcommands.
This feature provides a simple way to control whether a user can interrupt your script
whileit’srunning.
By default, when you run a script in a terminal session shell, the interactive shell is
suspended until the script completes. You can cause a script or command to run in
backgroundmodebyaddinganampersandsign(&)afterthecommandname.Whenyou
runascriptorcommandinbackgroundmode,theinteractiveshellreturns,allowingyou
to continue entering more commands. Any background processes run using this method
are still tied to the terminal session. If you exit the terminal session, the background
processesalsoexit.
To prevent this from happening, use the nohup command. This command intercepts any
signals intended for the command that would stop it — for example, when you exit the
terminalsession.Thisallowsscriptstocontinuerunninginbackgroundmodeevenifyou
exittheterminalsession.
Whenyoumoveaprocesstobackgroundmode,youcanstillcontrolwhathappenstoit.
Thejobscommandallowsyoutoviewprocessesstartedfromtheshellsession.Afteryou
knowthejobIDofabackgroundprocess,youcanusethe killcommandtosendLinux
signalstotheprocessorusethe fgcommandtobringtheprocessbacktotheforeground
in the shell session. You can suspend a running foreground process by using the Ctrl+Z
keycombinationandplaceitbackinbackgroundmode,usingthebgcommand.
The niceand renicecommandsallowyoutochangetheprioritylevelofaprocess.By
givingaprocessalowerpriority,youallowtheCPUtoallocatelesstimetoit.Thiscomes
inhandywhenrunninglongprocessesthatcantakelotsofCPUtime.
Inadditiontocontrollingprocesseswhilethey’rerunning,youcanalsodeterminewhena
processstartsonthesystem.Insteadofrunningascriptdirectlyfromthecommandline
interface prompt, you can schedule the process to run at an alternative time. You can
accomplish this in several different ways. The at command enables you to run a script
once at a preset time. The cron program provides an interface that can run scripts at a
regularlyscheduledinterval.
Finally,theLinuxsystemprovidesscriptfilesforyoutouseforschedulingyourscriptsto
runwheneverauserstartsanewbashshell.Similarly,thestartupfiles,suchas .bashrc,
are located in every user’s home directory to provide a location to place scripts and
commandsthatrunwithanewshell.
Inthenextchapter,welookathowtowritescriptfunctions.Scriptfunctionsallowyouto
writecodeblocksonceandthenusetheminmultiplelocationsthroughoutyourscript.
PartIII
AdvancedShellScripting
InThisPart
1. Chapter17CreatingFunctions
1. Chapter18WritingScriptsforGraphicalDesktops
1. Chapter19Introducingsedandgawk
1. Chapter20RegularExpressions
1. Chapter21Advancedsed
1. Chapter22Advancedgawk
1. Chapter23WorkingwithAlternativeShells
Chapter17
CreatingFunctions
InThisChapter
1. Basicscriptfunctions
2. Returningavalue
3. Usingvariablesinfunctions
4. Arrayandvariablefunctions
5. Functionrecursion
6. Creatingalibrary
7. Usingfunctionsonthecommandline
Oftenwhilewritingshellscripts,you’llfindyourselfusingthesamecodeinmultiple
locations.Ifit’sjustasmallcodesnippet,it’susuallynotthatbigofadeal.However,
rewritinglargechunksofcodemultipletimesinyourshellscriptcangettiring.The
bashshellprovidesawaytohelpyououtbysupportinguser-definedfunctions.You
canencapsulateyourshellscriptcodeintoafunctionanduseitasmanytimesasyou
wantanywhereinyourscript.Thischapterwalksyouthroughtheprocessofcreating
yourownshellscriptfunctionsanddemonstrateshowtousetheminothershell
scriptapplications.
BasicScriptFunctions
Asyoustartwritingmorecomplexshellscripts,you’llfindyourselfreusingpartsofcode
that perform specific tasks. Sometimes, it’s something simple, such as displaying a text
message and retrieving an answer from the script users. Other times, it’s a complicated
calculationthat’susedmultipletimesinyourscriptaspartofalargerprocess.
Ineachofthesesituations,itcangettiresomewritingthesameblocksofcodeoverand
overinyourscript.Itwouldbenicetojustwritetheblockofcodeonceandbeableto
refertothatblockofcodeanywhereinyourscriptwithouthavingtorewriteit.
The bash shell provides a feature allowing you to do just that. Functions are blocks of
scriptcodethatyouassignanametoandreuseanywhereinyourcode.Anytimeyouneed
tousethatblockofcodeinyourscript,yousimplyusethefunctionnameyouassignedit
(referredtoascallingthefunction).Thissectiondescribeshowtocreateandusefunctions
inyourshellscripts.
Creatingafunction
There are two formats you can use to create functions in bash shell scripts. The first
formatusesthekeyword function,alongwiththefunctionnameyouassigntotheblock
ofcode:
functionname{
commands
}
The name attribute defines a unique name assigned to the function. Each function you
defineinyourscriptmustbeassignedauniquename.
The commandsareoneormorebashshellcommandsthatmakeupyourfunction.When
you call the function, the bash shell executes each of the commands in the order they
appearinthefunction,justasinanormalscript.
Thesecondformatfordefiningafunctioninabashshellscriptmorecloselyfollowshow
functionsaredefinedinotherprogramminglanguages:
name(){
commands
}
The empty parentheses after the function name indicate that you’re defining a function.
Thesamenamingrulesapplyinthisformatasintheoriginalshellscriptfunctionformat.
Usingfunctions
Touseafunctioninyourscript,specifythefunctionnameonaline,justasyouwouldany
othershellcommand:
$cattest1
#!/bin/bash
#usingafunctioninascript
functionfunc1{
echo“Thisisanexampleofafunction”
}
count=1
while[$count-le5]
do
func1
count=$[$count+1]
done
echo“Thisistheendoftheloop”
func1
echo“Nowthisistheendofthescript”
$
$./test1
Thisisanexampleofafunction
Thisisanexampleofafunction
Thisisanexampleofafunction
Thisisanexampleofafunction
Thisisanexampleofafunction
Thisistheendoftheloop
Thisisanexampleofafunction
Nowthisistheendofthescript
$
Each time you reference the func1 function name, the bash shell returns to the func1
functiondefinitionandexecutesanycommandsyoudefinedthere.
The function definition doesn’t have to be the first thing in your shell script, but be
careful.Ifyouattempttouseafunctionbeforeit’sdefined,you’llgetanerrormessage:
$cattest2
#!/bin/bash
#usingafunctionlocatedinthemiddleofascript
count=1
echo“Thislinecomesbeforethefunctiondefinition”
functionfunc1{
echo“Thisisanexampleofafunction”
}
while[$count-le5]
do
func1
count=$[$count+1]
done
echo“Thisistheendoftheloop”
func2
echo“Nowthisistheendofthescript”
functionfunc2{
echo“Thisisanexampleofafunction”
}
$
$./test2
Thislinecomesbeforethefunctiondefinition
Thisisanexampleofafunction
Thisisanexampleofafunction
Thisisanexampleofafunction
Thisisanexampleofafunction
Thisisanexampleofafunction
Thisistheendoftheloop
./test2:func2:commandnotfound
Nowthisistheendofthescript
$
Thefirstfunction, func1,wasdefinedafteracoupleofstatementsinthescript,whichis
perfectlyfine.Whenthe func1 function was used in the script, the shell knew where to
findit.
However,thescriptattemptedtousethefunc2functionbeforeitwasdefined.Becausethe
func2 function wasn’t defined, when the script reached the place where we used it, it
producedanerrormessage.
You also need to be careful about your function names. Remember, each function name
mustbeunique,oryou’llhaveaproblem.Ifyouredefineafunction,thenewdefinition
overridestheoriginalfunctiondefinition,withoutproducinganyerrormessages:
$cattest3
#!/bin/bash
#testingusingaduplicatefunctionname
functionfunc1{
echo“Thisisthefirstdefinitionofthefunctionname”
}
func1
functionfunc1{
echo“Thisisarepeatofthesamefunctionname”
}
func1
echo“Thisistheendofthescript”
$
$./test3
Thisisthefirstdefinitionofthefunctionname
Thisisarepeatofthesamefunctionname
Thisistheendofthescript
$
Theoriginaldefinitionofthe func1functionworksfine,butaftertheseconddefinitionof
thefunc1function,anysubsequentusesofthefunctionusetheseconddefinition.
ReturningaValue
The bash shell treats functions like mini-scripts, complete with an exit status (see
Chapter11). There are three different ways you can generate an exit status for your
functions.
Thedefaultexitstatus
Bydefault,theexitstatusofafunctionistheexitstatusreturnedbythelastcommandin
thefunction.Afterthefunctionexecutes,youusethestandard$?variabletodeterminethe
exitstatusofthefunction:
$cattest4
#!/bin/bash
#testingtheexitstatusofafunction
func1(){
echo“tryingtodisplayanon-existentfile”
ls-lbadfile
}
echo“testingthefunction:“
func1
echo“Theexitstatusis:$?”
$
$./test4
testingthefunction:
tryingtodisplayanon-existentfile
ls:badfile:Nosuchfileordirectory
Theexitstatusis:1
$
The exit status of the function is 1 because the last command in the function failed.
However, you have no way of knowing if any of the other commands in the function
completedsuccessfullyornot.Lookatthisexample:
$cattest4b
#!/bin/bash
#testingtheexitstatusofafunction
func1(){
ls-lbadfile
echo“Thiswasatestofabadcommand”
}
echo“testingthefunction:”
func1
echo“Theexitstatusis:$?”
$
$./test4b
testingthefunction:
ls:badfile:Nosuchfileordirectory
Thiswasatestofabadcommand
Theexitstatusis:0
$
Thistime,becausethefunctionendedwithanechostatementthatcompletedsuccessfully,
the exit status of the function is 0, even though one of the commands in the function
failed.Usingthedefaultexitstatusofafunctioncanbeadangerouspractice.Fortunately,
wehaveacoupleofothersolutions.
Usingthereturncommand
Thebashshellusesthereturncommandtoexitafunctionwithaspecificexitstatus.The
returncommandallowsyoutospecifyasingleintegervaluetodefinethefunctionexit
status, providing an easy way for you to programmatically set the exit status of your
function:
$cattest5
#!/bin/bash
#usingthereturncommandinafunction
functiondbl{
read-p“Enteravalue:”value
echo“doublingthevalue”
return$[$value*2]
}
dbl
echo“Thenewvalueis$?”
$
The dblfunctiondoublestheintegervaluecontainedinthe $valuevariableprovidedby
the user input. It then returns the result using the return command, which the script
displaysusingthe$?variable.
Youmustbecareful,however,whenusingthistechniquetoreturnavaluefromafunction.
Keepthefollowingtwotipsinmindtoavoidproblems:
Remembertoretrievethereturnvalueassoonasthefunctioncompletes.
Rememberthatanexitstatusmustbeintherangeof0to255.
Ifyouexecuteanyothercommandsbeforeretrievingthevalueofthefunction,usingthe
$? variable, the return value from the function is lost. Remember that the $? variable
returnstheexitstatusofthelastexecutedcommand.
Thesecondproblemdefinesalimitationforusingthisreturnvaluetechnique.Becausean
exitstatusmustbelessthan256,theresultofyourfunctionmustproduceanintegervalue
lessthan256.Anyvalueoverthatreturnsanerrorvalue:
$./test5
Enteravalue:200
doublingthevalue
Thenewvalueis1
$
Youcannotusethisreturnvaluetechniqueifyouneedtoreturneitherlargerintegervalues
or a string value. Instead, you need to use another method, demonstrated in the next
section.
Usingfunctionoutput
Justasyoucancapturetheoutputofacommandtoashellvariable,youcanalsocapture
theoutputofafunctiontoashellvariable.Youcanusethistechniquetoretrieveanytype
ofoutputfromafunctiontoassigntoavariable:
result=‘dbl’
Thiscommandassignstheoutputofthedblfunctiontothe$resultshellvariable.Here’s
anexampleofusingthismethodinascript:
$cattest5b
#!/bin/bash
#usingtheechotoreturnavalue
functiondbl{
read-p“Enteravalue:”value
echo$[$value*2]
}
result=$(dbl)
echo“Thenewvalueis$result”
$
$./test5b
Enteravalue:200
Thenewvalueis400
$
$./test5b
Enteravalue:1000
Thenewvalueis2000
$
Thenewfunctionnowusesan echostatementtodisplaytheresultofthecalculation.The
scriptjustcapturestheoutputofthe dblfunctioninsteadoflookingattheexitstatusfor
theanswer.
There’sasubtletrickthatthisexampledemonstrates.You’llnoticethatthe db1function
really outputs two messages. The read command outputs a short message querying the
userforthevalue.Thebashshellscriptissmartenoughtonotconsiderthisaspartofthe
STDOUT output and ignores it. If you had used an echo statement to produce this query
messagetotheuser,itwouldhavebeencapturedbytheshellvariableaswellastheoutput
value.
Note
Usingthistechnique,youcanalsoreturnfloatingpointandstringvalues,makingthis
anextremelyversatilemethodforreturningvaluesfromfunctions.
UsingVariablesinFunctions
You might have noticed in the test5 example in the previous section that we used a
variablecalled $valuewithinthefunctiontoholdthevaluethatitprocessed.Whenyou
use variables in your functions, you need to be somewhat careful about how you define
andhandlethem.Thisisacommoncauseofproblemsinshellscripts.Thissectiongoes
over a few techniques for handling variables both inside and outside your shell script
functions.
Passingparameterstoafunction
Asmentionedearlierinthe“ReturningaValue”section,thebashshelltreatsfunctionsjust
likemini-scripts.Thismeansthatyoucanpassparameterstoafunctionjustlikearegular
script(seeChapter14).
Functions can use the standard parameter environment variables to represent any
parameters passed to the function on the command line. For example, the name of the
functionisdefinedinthe $0variable,andanyparametersonthefunctioncommandline
aredefinedusingthevariables $1,$2,andsoon.Youcanalsousethespecialvariable $#
todeterminethenumberofparameterspassedtothefunction.
Whenspecifyingthefunctioninyourscript,youmustprovidetheparametersonthesame
commandlineasthefunction,likethis:
func1$value110
The function can then retrieve the parameter values using the parameter environment
variables.Here’sanexampleofusingthismethodtopassvaluestoafunction:
$cattest6
#!/bin/bash
#passingparameterstoafunction
functionaddem{
if[$#-eq0]||[$#-gt2]
then
echo-1
elif[$#-eq1]
then
echo$[$1+$1]
else
echo$[$1+$2]
fi
}
echo-n“Adding10and15:“
value=$(addem1015)
echo$value
echo-n“Let’stryaddingjustonenumber:“
value=$(addem10)
echo$value
echo-n“Nowtryingaddingnonumbers:“
value=$(addem)
echo$value
echo-n“Finally,tryaddingthreenumbers:“
value=$(addem101520)
echo$value
$
$./test6
Adding10and15:25
Let’stryaddingjustonenumber:20
Nowtryingaddingnonumbers:-1
Finally,tryaddingthreenumbers:-1
$
The addemfunctioninthe text6scriptfirstchecksthenumberofparameterspassedtoit
by the script. If there aren’t any parameters, or if there are more than two parameters,
addem returns a value of -1. If there’s just one parameter, addem adds the parameter to
itselffortheresult.Iftherearetwoparameters,addemaddsthemtogetherfortheresult.
Because the function uses the special parameter environment variables for its own
parameter values, it can’t directly access the script parameter values from the command
lineofthescript.Thefollowingexamplefails:
$catbadtest1
#!/bin/bash
#tryingtoaccessscriptparametersinsideafunction
functionbadfunc1{
echo$[$1*$2]
}
if[$#-eq2]
then
value=$(badfunc1)
echo“Theresultis$value”
else
echo“Usage:badtest1ab”
fi
$
$./badtest1
Usage:badtest1ab
$./badtest11015
./badtest1:*:syntaxerror:operandexpected(errortokenis“*
“)
Theresultis
$
Even though the function uses the $1 and $2 variables, they aren’t the same $1 and $2
variablesavailableinthemainpartofthescript.Instead,ifyouwanttousethosevaluesin
yourfunction,youhavetomanuallypassthemwhenyoucallthefunction:
$cattest7
#!/bin/bash
#tryingtoaccessscriptparametersinsideafunction
functionfunc7{
echo$[$1*$2]
}
if[$#-eq2]
then
value=$(func7$1$2)
echo“Theresultis$value”
else
echo“Usage:badtest1ab”
fi
$
$./test7
Usage:badtest1ab
$./test71015
Theresultis150
$
Bypassingthe $1and$2variablestothefunction,theybecomeavailableforthefunction
touse,justlikeanyotherparameter.
Handlingvariablesinafunction
One thing that causes problems for shell script programmers is the scope of a variable.
The scope is where the variable is visible. Variables defined in functions can have a
different scope than regular variables. That is, they can be hidden from the rest of the
script.
Functionsusetwotypesofvariables:
Global
Local
Thefollowingsectionsdescribehowtousebothtypesofvariablesinyourfunctions.
Globalvariables
Globalvariablesarevariablesthatarevalidanywherewithintheshellscript.Ifyoudefine
a global variable in the main section of a script, you can retrieve its value inside a
function.Likewise,ifyoudefineaglobalvariableinsideafunction,youcanretrieveits
valueinthemainsectionofthescript.
Bydefault,anyvariablesyoudefineinthescriptareglobalvariables.Variablesdefined
outsideofafunctioncanbeaccessedwithinthefunctionjustfine:
$cattest8
#!/bin/bash
#usingaglobalvariabletopassavalue
functiondbl{
value=$[$value*2]
}
read-p“Enteravalue:”value
dbl
echo“Thenewvalueis:$value”
$
$./test8
Enteravalue:450
Thenewvalueis:900
$
The$valuevariableisdefinedoutsideofthefunctionandassignedavalueoutsideofthe
function.Whenthe dblfunctioniscalled,thevariableanditsvaluearestillvalidinside
thefunction.Whenthevariableisassignedanewvalueinsidethefunction,thatnewvalue
isstillvalidwhenthescriptreferencesthevariable.
Thiscanbeadangerouspractice,however,especiallyifyouintendtouseyourfunctions
indifferentshellscripts.Itrequiresthatyouknowexactlywhatvariablesareusedinthe
function,includinganyvariablesusedtocalculatevaluesnotreturnedtothescript.Here’s
anexampleofhowthingscangobad:
$catbadtest2
#!/bin/bash
#demonstratingabaduseofvariables
functionfunc1{
temp=$[$value+5]
result=$[$temp*2]
}
temp=4
value=6
func1
echo“Theresultis$result”
if[$temp-gt$value]
then
echo“tempislarger”
else
echo“tempissmaller”
fi
$
$./badtest2
Theresultis22
tempislarger
$
Because the $temp variable was used in the function, its value is compromised in the
script, producing a result that you may not have intended. There’s an easy way to solve
thisprobleminyourfunctions,asshowninthenextsection.
Localvariables
Instead of using global variables in functions, any variables that the function uses
internally can be declared as local variables. To do that, just use the local keyword in
frontofthevariabledeclaration:
localtemp
Youcanalsousethelocalkeywordinanassignmentstatementwhileassigningavalueto
thevariable:
localtemp=$[$value+5]
The local keyword ensures that the variable is limited to only within the function. If a
variablewiththesamenameappearsoutsidethefunctioninthescript,theshellkeepsthe
two variable values separate. Now you can easily keep your function variables separate
fromyourscriptvariablesandshareonlytheonesyouwanttoshare:
$cattest9
#!/bin/bash
#demonstratingthelocalkeyword
functionfunc1{
localtemp=$[$value+5]
result=$[$temp*2]
}
temp=4
value=6
func1
echo“Theresultis$result”
if[$temp-gt$value]
then
echo“tempislarger”
else
echo“tempissmaller”
fi
$
$./test9
Theresultis22
tempissmaller
$
Now when you use the $temp variable within the func1 function, it doesn’t affect the
valueassignedtothe$tempvariableinthemainscript.
ArrayVariablesandFunctions
Chapter6discussedanadvancedwayofallowingasinglevariabletoholdmultiplevalues
byusingarrays.Usingarrayvariablevalueswithfunctionsisalittletricky,andthereare
somespecialconsiderations.Thissectiondescribesatechniquethatallowsyoutodothat.
Passingarraystofunctions
Theartofpassinganarrayvariabletoascriptfunctioncanbeconfusing.Ifyoutrytopass
thearrayvariableasasingleparameter,itdoesn’twork:
$catbadtest3
#!/bin/bash
#tryingtopassanarrayvariable
functiontestit{
echo“Theparametersare:$@”
thisarray=$1
echo“Thereceivedarrayis${thisarray[*]}”
}
myarray=(12345)
echo“Theoriginalarrayis:${myarray[*]}”
testit$myarray
$
$./badtest3
Theoriginalarrayis:12345
Theparametersare:1
Thereceivedarrayis1
$
Ifyoutryusingthearrayvariableasafunctionparameter,thefunctiononlypicksupthe
firstvalueofthearrayvariable.
Tosolvethisproblem,youmustdisassemblethearrayvariableintoitsindividualvalues
andusethevaluesasfunctionparameters.Insidethefunction,youcanreassembleallthe
parametersintoanewarrayvariable.Here’sanexampleofdoingthis:
$cattest10
#!/bin/bash
#arrayvariabletofunctiontest
functiontestit{
localnewarray
newarray=(;‘echo“$@”’)
echo“Thenewarrayvalueis:${newarray[*]}”
}
myarray=(12345)
echo“Theoriginalarrayis${myarray[*]}”
testit${myarray[*]}
$
$./test10
Theoriginalarrayis12345
Thenewarrayvalueis:12345
$
Thescriptusesthe$myarrayvariabletoholdalltheindividualarrayvaluestoplacethem
all on the command line for the function. The function then rebuilds the array variable
fromthecommandlineparameters.Onceinsidethefunction,thearraycanbeusedjust
likeanyotherarray:
$cattest11
#!/bin/bash
#addingvaluesinanarray
functionaddarray{
localsum=0
localnewarray
newarray=($(echo“$@”))
forvaluein${newarray[*]}
do
sum=$[$sum+$value]
done
echo$sum
}
myarray=(12345)
echo“Theoriginalarrayis:${myarray[*]}”
arg1=$(echo${myarray[*]})
result=$(addarray$arg1)
echo“Theresultis$result”
$
$./test11
Theoriginalarrayis:12345
Theresultis15
$
The addarray function iterates through the array values, adding them together. You can
putanynumberofvaluesinthe myarrayarrayvariable,andthe addarrayfunctionadds
them.
Returningarraysfromfunctions
Passinganarrayvariablefromafunctionbacktotheshellscriptusesasimilartechnique.
The function uses an echo statement to output the individual array values in the proper
order,andthescriptmustreassemblethemintoanewarrayvariable:
$cattest12
#!/bin/bash
#returninganarrayvalue
functionarraydblr{
localorigarray
localnewarray
localelements
locali
origarray=($(echo“$@”))
newarray=($(echo“$@”))
elements=$[$#-1]
for((i=0;i<=$elements;i++))
{
newarray[$i]=$[${origarray[$i]}*2]
}
echo${newarray[*]}
}
myarray=(12345)
echo“Theoriginalarrayis:${myarray[*]}”
arg1=$(echo${myarray[*]})
result=($(arraydblr$arg1))
echo“Thenewarrayis:${result[*]}”
$
$./test12
Theoriginalarrayis:12345
Thenewarrayis:246810
Thescriptpassesthearrayvalue,usingthe$arg1variabletothearraydblrfunction.The
arraydblrfunctionreassemblesthearrayintoanewarrayvariable,anditmakesacopy
fortheoutputarrayvariable.Ittheniteratesthroughtheindividualarrayvariablevalues,
doubleseachvalue,andplacesitintothecopyofthearrayvariableinthefunction.
Thearraydblrfunctionthenusestheechostatementtooutputtheindividualvaluesofthe
arrayvariablevalues.Thescriptusestheoutputofthearraydblrfunctiontoreassemblea
newarrayvariablewiththevalues.
FunctionRecursion
One feature that local function variables provide is self-containment. A self-contained
functiondoesn’tuseanyresourcesoutsideofthefunction,otherthanwhatevervariables
thescriptpassestoitinthecommandline.
Thisfeatureenablesthefunctiontobecalledrecursively,whichmeansthatthefunction
calls itself to reach an answer. Usually, a recursive function has a base value that it
eventually iterates down to. Many advanced mathematical algorithms use recursion to
reduceacomplexequationdownonelevelrepeatedly,untiltheygettotheleveldefined
bythebasevalue.
The classic example of a recursive algorithm is calculating factorials. A factorial of a
numberisthevalueoftheprecedingnumbersmultipliedwiththenumber.Thus,tofind
thefactorialof5,you’dperformthefollowingequation:
5!=1*2*3*4*5=120
Usingrecursion,theequationisreduceddowntothefollowingformat:
x!=x*(x-1)!
or in English, the factorial of x is equal to x times the factorial of x-1. This can be
expressedinasimplerecursivescript:
functionfactorial{
if[$1-eq1]
then
echo1
else
localtemp=$[$1-1]
localresult=‘factorial$temp’
echo$[$result*$1]
fi
}
Thefactorialfunctionusesitselftocalculatethevalueforthefactorial:
$cattest13
#!/bin/bash
#usingrecursion
functionfactorial{
if[$1-eq1]
then
echo1
else
localtemp=$[$1-1]
localresult=$(factorial$temp)
echo$[$result*$1]
fi
}
read-p“Entervalue:”value
result=$(factorial$value)
echo“Thefactorialof$valueis:$result”
$
$./test13
Entervalue:5
Thefactorialof5is:120
$
Usingthefactorialfunctioniseasy.Havingcreatedafunctionlikethis,youmaywantto
useitinotherscripts.Next,welookathowtodothatefficiently.
CreatingaLibrary
It’seasytoseehowfunctionscanhelpsavetypinginasinglescript,butwhatifyoujust
happen to use the same single code block between scripts? It’s obviously challenging if
youhavetodefinethesamefunctionineachscript,onlytouseitonetimeineachscript.
There’sasolutionforthatproblem!Thebashshellallowsyoutocreatealibraryfilefor
yourfunctionsandthenreferencethatsinglelibraryfileinasmanyscriptsasyouneedto.
Thefirststepintheprocessistocreateacommonlibraryfilethatcontainsthefunctions
you need in your scripts. Here’s a simple library file called myfuncs that defines three
simplefunctions:
$catmyfuncs
#myscriptfunctions
functionaddem{
echo$[$1+$2]
}
functionmultem{
echo$[$1*$2]
}
functiondivem{
if[$2-ne0]
then
echo$[$1/$2]
else
echo-1
fi
}
$
Thenextstepistoincludethemyfuncslibraryfileinyourscriptfilesthatwanttouseany
ofthefunctions.Thisiswherethingsgettricky.
The problem is with the scope of shell functions. As with environment variables, shell
functions are valid only for the shell session in which you define them. If you run the
myfuncs shell script from your shell command line interface prompt, the shell creates a
new shell and runs the script in that new shell. This defines the three functions for that
shell, but when you try to run another script that uses those functions, they aren’t
available.
Thisappliestoscriptsaswell.Ifyoutrytojustrunthelibraryfileasaregularscriptfile,
thefunctionsdon’tappearinyourscript:
$catbadtest4
#!/bin/bash
#usingalibraryfilethewrongway
./myfuncs
result=$(addem1015)
echo“Theresultis$result”
$
$./badtest4
./badtest4:addem:commandnotfound
Theresultis
$
The key to using function libraries is the source command. The source command
executes commands within the current shell context instead of creating a new shell to
execute them. You use the source command to run the library file script inside of your
shellscript.Thismakesthefunctionsavailabletothescript.
Thesourcecommandhasashortcutalias,calledthedotoperator.Tosourcethe myfuncs
libraryfileinashellscript,youjustneedtoaddthefollowingline:
../myfuncs
Thisexampleassumesthatthe myfuncslibraryfileislocatedinthesamedirectoryasthe
shell script. If not, you need to use the appropriate path to access the file. Here’s an
exampleofcreatingascriptthatusesthemyfuncslibraryfile:
$cattest14
#!/bin/bash
#usingfunctionsdefinedinalibraryfile
../myfuncs
value1=10
value2=5
result1=$(addem$value1$value2)
result2=$(multem$value1$value2)
result3=$(divem$value1$value2)
echo“Theresultofaddingthemis:$result1”
echo“Theresultofmultiplyingthemis:$result2”
echo“Theresultofdividingthemis:$result3”
$
$./test14
Theresultofaddingthemis:15
Theresultofmultiplyingthemis:50
Theresultofdividingthemis:2
$
Thescriptsuccessfullyusesthefunctionsdefinedinthemyfuncslibraryfile.
UsingFunctionsontheCommandLine
You can use script functions to create some pretty complex operations. Sometimes, it
would be nice to be able to use these functions directly on the command line interface
prompt.
Justasyoucanuseascriptfunctionasacommandinashellscript,youcanalsousea
scriptfunctionasacommandinthecommandlineinterface.Thisisanicefeaturebecause
afteryoudefinethefunctionintheshell,youcanuseitfromanydirectoryonthesystem;
youdon’thavetoworryaboutascriptbeinginyourPATHenvironmentvariable.Thetrick
istogettheshelltorecognizethefunction.Youcandothatinacoupleofways.
Creatingfunctionsonthecommandline
Because the shell interprets commands as you type them, you can define a function
directlyonthecommandline.Youcandothatintwoways.
Thefirstmethoddefinesthefunctionallononeline:
$functiondivem{echo$[$1/$2];}
$divem1005
20
$
When you define the function on the command line, you must remember to include a
semicolonattheendofeachcommand,sotheshellknowswheretoseparatecommands:
$functiondoubleit{read-p“Entervalue:”value;echo$[
$value*2];}
$
$doubleit
Entervalue:20
40
$
The other method is to use multiple lines to define the function. When you do that, the
bash shell uses the secondary prompt to prompt you for more commands. Using this
method,youdon’tneedtoplaceasemicolonattheendofeachcommand;justpressthe
Enterkey:
$functionmultem{
>echo$[$1*$2]
>}
$multem25
10
$
When you use the brace at the end of the function, the shell knows that you’re finished
definingthefunction.
Caution
Beextremelycarefulwhencreatingfunctionsonthecommandline.Ifyouusea
functionwiththesamenameasabuilt-incommandoranothercommand,the
functionoverridestheoriginalcommand.
Definingfunctionsinthe.bashrcfile
The obvious downside to defining shell functions directly on the command line is that
whenyouexittheshell,yourfunctiondisappears.Forcomplexfunctions,thiscanbecome
aproblem.
A much simpler method is to define the function in a place where it is reloaded by the
shelleachtimeyoustartanewshell.
Thebestplacetodothatisthe.bashrcfile.Thebashshelllooksforthisfileinyourhome
directoryeachtimeitstarts,whetherinteractivelyorastheresultofstartinganewshell
fromwithinanexistingshell.
Directlydefiningfunctions
You can define the functions directly in the .bashrc file in your home directory. Most
Linux distributions already define some things in the .bashrc file, so be careful not to
removethoseitems.Justaddyourfunctionstothebottomoftheexistingfile.Here’san
exampleofdoingthat:
$cat.bashrc
#.bashrc
#Sourceglobaldefinitions
if[-r/etc/bashrc];then
./etc/bashrc
fi
functionaddem{
echo$[$1+$2]
}
$
Thefunctiondoesn’ttakeeffectuntilthenexttimeyoustartanewbashshell.Afteryou
dothat,youcanusethefunctionanywhereonthesystem.
Sourcingfunctionfiles
Justasinashellscript,youcanusethe sourcecommand(oritsaliasthedotoperator)to
addfunctionsfromanexistinglibraryfiletoyour.bashrcscript:
$cat.bashrc
#.bashrc
#Sourceglobaldefinitions
if[-r/etc/bashrc];then
./etc/bashrc
fi
./home/rich/libraries/myfuncs
$
Makesurethatyouincludetheproperpathnametoreferencethelibraryfileforthebash
shelltofind.Thenexttimeyoustartashell,allthefunctionsinyourlibraryareavailable
atthecommandlineinterface:
$addem105
15
$multem105
50
$divem105
2
$
Even better, the shell also passes any defined functions to child shell processes so your
functionsareautomaticallyavailableforanyshellscriptsyourunfromyourshellsession.
Youcantestthisbywritingascriptthatusesthefunctionswithoutdefiningorsourcing
them:
$cattest15
#!/bin/bash
#usingafunctiondefinedinthe.bashrcfile
value1=10
value2=5
result1=$(addem$value1$value2)
result2=$(multem$value1$value2)
result3=$(divem$value1$value2)
echo“Theresultofaddingthemis:$result1”
echo“Theresultofmultiplyingthemis:$result2”
echo“Theresultofdividingthemis:$result3”
$
$./test15
Theresultofaddingthemis:15
Theresultofmultiplyingthemis:50
Theresultofdividingthemis:2
$
Evenwithoutsourcingthelibraryfile,thefunctionsworkedperfectlyintheshellscript.
FollowingaPracticalExample
There’smuchmoretousingfunctionsthanjustcreatingyourownfunctionstoworkwith.
In the open source world, code sharing is key, and that also applies to shell script
functions. Quite a few different shell script functions are available for you to download
anduseinyourownapplications.
Thissectionwalksthroughdownloading,installing,andusingtheGNUshtoolshellscript
function library. The shtool library provides some simple shell script functions for
performingeverydayshellfunctions,suchasworkingwithtemporaryfilesandfoldersor
formattingoutputtodisplay.
Downloadingandinstalling
The first step in the process is to download and install the GNU shtool library to your
systemsoyoucanusethelibraryfunctionsinyourownshellscripts.Todothat,youneed
to use an FTP client program or a browser in a graphical desktop. Use this URL to
downloadtheshtoolpackage:
ftp://ftp.gnu.org/gnu/shtool/shtool-2.0.8.tar.gz
This downloads the file shtool-2.0.8.tar.gz to the download folder. From there, you
can use the cp command line tool or the graphical file manager tool in your Linux
distribution(suchasNautilusinUbuntu)tocopythefiletoyourHomefolder.
AfteryoucopythefiletoyourHomefolder,youcanextractitusingthetarcommand:
tar-zxvfshtool-2.0.8.tar.gz
This extracts the package files into a folder named shtool-2.0.8.Nowyou’rereadyto
buildtheshellscriptlibraryfile.
Buildingthelibrary
TheshtooldistributionfilemustbeconfiguredforyourspecificLinuxenvironment.Todo
that, it uses standard configure and make commands, commonly used in the C
programmingenvironment.Tobuildthelibraryfile,youjustneedtoruntwocommands:
$./confifgure
$make
Theconfigurecommandchecksthesoftwarenecessarytobuildtheshtoollibraryfile.As
it finds the tools it needs, it modifies the configuration file with the proper paths to the
tools.
Themakecommandrunsthroughthestepstobuildtheshtoollibraryfile.Theresultingfile
(shtool) is the full library package file. You can test the library file using the make
commandaswell:
$maketest
Runningtestsuite:
echo………..ok
mdate……….ok
table……….ok
prop………..ok
move………..ok
install……..ok
mkdir……….ok
mkln………..ok
mkshadow…….ok
fixperm……..ok
rotate………ok
tarball……..ok
subst……….ok
platform…….ok
arx…………ok
slo…………ok
scpp………..ok
version……..ok
path………..ok
OK:passed:19/19
$
Thetestmodetestsallthefunctionsavailableintheshtoollibrary.Ifallpass,thenyou’re
ready to install the library into a common location on your Linux system so all your
scripts can use it. To do that, you can use the install option of the make command.
However,youneedtobeloggedinastherootuseraccounttorunit:
$su
Password:
#makeinstall
./shtoolmkdir-f-p-m755/usr/local
./shtoolmkdir-f-p-m755/usr/local/bin
./shtoolmkdir-f-p-m755/usr/local/share/man/man1
./shtoolmkdir-f-p-m755/usr/local/share/aclocal
./shtoolmkdir-f-p-m755/usr/local/share/shtool
…
./shtoolinstall-c-m644sh.version/usr/local/share/shtool/sh.version
./shtoolinstall-c-m644sh.path/usr/local/share/shtool/sh.path
#
Nowyou’rereadytostartusingthefunctionsinyourownshellscripts!
Theshtoollibraryfunctions
Theshtoollibraryprovidesquiteafewfunctionsthatcancomeinhandywhenworking
withshellscripts.Table17.1showsthefunctionsavailableinthelibrary.
Table17.1TheshtoolLibraryFunctions
Function
Description
Arx
Createsanarchivewithextendedfeatures
Echo Displaysthestringvaluewithconstructexpansion
fixperm
Changesfilepermissionsinsideafoldertree
install
Installsascriptorfile
mdate
Displaysmodificationtimeofafileordirectory
mkdir
Createsoneormoredirectories
Mkln
Createsalinkusingrelativepaths
mkshadow
move
Path
platform
Prop
rotate
Scpp
Slo
Subst
Table
tarball
version
Createsashadowtree
Movesfileswithsubstitution
Workswithprogrampaths
Displaystheplatformidentity
Displaysananimatedprogresspropeller
Rotateslogfiles
ThesharingCpre-processor
Separateslinkeroptionsbylibraryclass
Usessedsubstitutionoperations
Displaysfield-separateddatainatableformat
Createstarfilesfromfilesandfolders
Createsaversioninformationfile
Eachoftheshtoolfunctionshaslotsofoptionsandargumentsthatyoucanusetomodify
howitworks.Here’stheformattouseashtoolfunction:
shtool[options][function[options][args]]
Usingthelibrary
You can use the shtool functions directly from the command line or from within your
shellscripts.Here’sanexampleofusingtheplatformfunctioninsideashellscript:
$cattest16
#!/bin/bash
shtoolplatform
$./test16
Ubuntu14.04(iX86)
$
The platformfunctionreturnstheLinuxdistributionandtheCPUhardwarethatthehost
systemisusing.Oneofmyfavoritesisthe propfunction.Itcreatesaspinningpropeller
fromalternatingthe∖,|,/,and–characterswhilesomethingisprocessing.That’sagreat
tool to help show your shell script users that something is happening in the background
whilethescriptisrunning.
Tousethe propfunction,youjustpipetheoutputofthefunctionyouwanttomonitorto
theshtoolscript:
$ls–al/usr/bin|shtoolprop–p“waiting…”
waiting…
$
Thepropfunctionalternatesbetweenthepropellercharacterstoindicatethatsomethingis
happening.Inthiscase,it’stheoutputfromthe lscommand.Howmuchofthatyousee
dependsonhowfastyourCPUcanlistoutallthefilesinthe /usr/binfolder!The –p
optionallowsyoutocustomizetheoutputtextthatappearsbeforethepropellercharacters.
Nowthat’sgettingfancy!
Summary
Shellscriptfunctionsallowyoutoplacescriptcodethat’srepeatedthroughoutthescript
in a single place. Instead of having to rewrite blocks of code, you can create a function
containing the code block and then just reference the function name in your script. The
bashshelljumpstothefunctioncodeblockwheneveritseesthefunctionnameusedinthe
script.
Youcanevencreatescriptfunctionsthatreturnvalues.Thisallowsyoutocreatefunctions
that interact with the script, returning both numeric and character data. Script functions
can return numeric data by using the exit status of the last command in the function or
usingthereturncommand.Thereturncommandallowsyoutoprogrammaticallysetthe
exitstatusofyourfunctiontoaspecificvaluebasedontheresultsofthefunction.
Functionscanalsoreturnvaluesusingthestandard echostatement.Youcancapturethe
output data using the backtick character as you would any other shell command. This
enablesyoutoreturnanytypeofdatafromafunction,includingstringsandfloating-point
numbers.
You can use shell variables within your functions, assigning values to variables and
retrieving values from existing variables. This allows you to pass any type of data both
intoandoutofascriptfunctionfromthemainscriptprogram.Functionsalsoallowyouto
define local variables, which are accessible only from within the function code block.
Local variables allow you to create self-contained functions, which don’t interfere with
anyvariablesorprocessesusedinthemainshellscript.
Functionscanalsocallotherfunctions,includingthemselves.Whenafunctioncallsitself,
itiscalledrecursion.Arecursivefunctionoftenhasabasevaluethatistheterminalvalue
of the function. The function continues to call itself with a decreasing parameter value
untilthebasevalueisreached.
If you use lots of functions in your shell scripts, you can create library files of script
functions. The library files can be included in any shell script file by using the source
command,oritsalias,thedotoperator.Thisiscalledsourcingthelibraryfile.Theshell
doesn’trunthelibraryfilebutmakesthefunctionsavailablewithintheshellthatrunsthe
script.Youcanusethissametechniquetocreatefunctionsthatyoucanuseonthenormal
shellcommandline.Youcaneitherdefinefunctionsdirectlyonthecommandlineoryou
can add them to your .bashrc file so they are available for each new shell session you
start. This is a handy way to create utilities that can be used no matter what your PATH
environmentvariableissetto.
Thenextchapterdiscussestheuseoftextgraphicsinyourscripts.Inthisdayofmodern
graphical interfaces, sometimes a plain text interface just doesn’t cut it. The bash shell
providessomeeasywaysforyoutoincorporatesimplegraphicsfeaturesinyourscriptsto
helpspicethingsup.
Chapter18
WritingScriptsforGraphicalDesktops
InThisChapter
1. Creatingtextmenus
2. Buildingtextwindowwidgets
3. AddingXWindowgraphics
Overtheyears,shellscriptshaveacquiredareputationforbeingdullandboring.
Thisdoesn’thavetobethecase,however,ifyouplanonrunningyourscriptsina
graphicalenvironment.Thereareplentyofwaystointeractwithyourscriptuserthat
don’trelyonthereadandechostatements.Thischapterdivesintoafewdifferent
methodsyoucanusetohelpaddlifetoyourinteractivescriptssotheydon’tlookso
old-fashioned.
CreatingTextMenus
Themostcommonwaytocreateaninteractiveshellscriptistoutilizeamenu.Offering
your customers a choice of various options helps guide them through exactly what the
scriptcanandcan’tdo.
Menuscriptsusuallyclearthedisplayareaandthenshowalistofoptionsavailable.The
customercanselectanoptionbypressinganassociatedletterornumberassignedtoeach
option.Figure18.1showsthelayoutofasamplemenu.
Figure18.1Displayingamenufromashellscript
The core of a shell script menu is the case command (see Chapter 12). The case
command performs specific commands, depending on what character your customer
selectsfromthemenu.
The following sections walk you through the steps you should follow to create a menubasedshellscript.
Createthemenulayout
The first step in creating a menu is, obviously, to determine what elements you want to
appearinthemenuandlaythemoutthewaythatyouwantthemtoappear.
Before creating the menu, it’s usually a good idea to clear the monitor display. This
enablesyoutodisplayyourmenuinacleanenvironmentwithoutdistractingtext.
The clear command uses the terminfo data of your terminal session (see Chapter 2) to
clearanytextthatappearsonthemonitor.Aftertheclearcommand,youcanusetheecho
commandtodisplayyourmenuelements.
Bydefault,the echocommandcanonlydisplayprintabletextcharacters.Whencreating
menu items, it’s often helpful to use nonprintable items, such as the tab and newline
characters. To include these characters in your echo command, you must use the -e
option.Thus,thecommand:
echo-e“1.\tDisplaydiskspace”
resultsintheoutputline:
1.Displaydiskspace
This greatly helps in formatting the layout of the menu items. With just a few echo
commands,youcancreateareasonable-lookingmenu:
clear
echo
echo-e“\t\t\tSysAdminMenu\n”
echo-e“\t1.Displaydiskspace”
echo-e“\t2.Displayloggedonusers”
echo-e“\t3.Displaymemoryusage”
echo-e“\t0.Exitmenu\n\n”
echo–en“\t\tEnteroption:“
The -enoptiononthelastlinedisplaysthelinewithoutaddingthenewlinecharacterat
theend.Thisgivesthemenuamoreprofessionallook,becausethecursorstaysattheend
ofthelinewaitingforthecustomer’sinput.
Thelastpartofcreatingthemenuistoretrievetheinputfromthecustomer.Thisisdone
usingthereadcommand(seeChapter14).Becauseweexpectonlysingle-characterinput,
the nice thing to do is to use the -n option in the read command to retrieve only one
character. This allows the customer to enter a number without having to press the Enter
key:
read-n1option
Next,youneedtocreateyourmenufunctions.
Createthemenufunctions
Shell script menu options are easier to create as a group of separate functions. This
enablesyoutocreateasimple,concisecasecommandthatiseasytofollow.
Todothat,youneedtocreateseparateshellfunctionsforeachofyourmenuoptions.The
first step in creating a menu shell script is to determine what functions you want your
scripttoperformandlaythemoutasseparatefunctionsinyourcode.
Itiscommonpracticetocreatestubfunctionsforfunctionsthataren’timplementedyet.A
stubfunctionisafunctionthatdoesn’tcontainanycommandsyetorpossiblyjustanecho
statementindicatingwhatshouldbethereeventually:
functiondiskspace{
clear
echo“Thisiswherethediskspacecommandswillgo”
}
Thisenablesyourmenutooperatesmoothlywhileyouworkontheindividualfunctions.
You don’t have to code all the functions for your menu to work. You’ll notice that the
functionstartsoutwiththe clearcommand.Thisenablesyoutostartthefunctionona
cleanmonitorscreen,withoutthemenushowing.
One thing that helps out in the shell script menu is to create the menu layout itself as a
function:
functionmenu{
clear
echo
echo-e“\t\t\tSysAdminMenu\n”
echo-e“\t1.Displaydiskspace”
echo-e“\t2.Displayloggedonusers”
echo-e“\t3.Displaymemoryusage”
echo-e“\t0.Exitprogram\n\n”
echo-en“\t\tEnteroption:“
read-n1option
}
Thisenablesyoutoeasilyredisplaythemenuatanytimejustbycallingthemenufunction.
Addthemenulogic
Now that you have your menu layout and your functions, you just need to create the
programming logic to put the two together. As mentioned, this requires the case
command.
The case command should call the appropriate function according to the character
selection expected from the menu. It’s always a good idea to use the default case
commandcharacter(theasterisk)tocatchanyincorrectmenuentries.
Thefollowingcodeillustratestheuseofthecasecommandinatypicalmenu:
menu
case$optionin
0)
break;;
1)
diskspace;;
2)
whoseon;;
3)
memusage;;
*)
clear
echo“Sorry,wrongselection”;;
esac
Thiscodefirstusesthe menufunctiontoclearthemonitorscreenanddisplaythemenu.
The readcommandinthe menufunctionpausesuntilthecustomerhitsacharacteronthe
keyboard.Afterthat’sbeendone,thecasecommandtakesover.Thecasecommandcalls
theappropriatefunctionbasedonthereturnedcharacter.Afterthefunctioncompletes,the
casecommandexits.
Puttingitalltogether
Nowthatyou’veseenallthepartsthatmakeupashellscriptmenu,let’sputthemtogether
andseehowtheyallinteroperate.Here’sanexampleofafullmenuscript:
$catmenu1
#!/bin/bash
#simplescriptmenu
functiondiskspace{
clear
df-k
}
functionwhoseon{
clear
who
}
functionmemusage{
clear
cat/proc/meminfo
}
functionmenu{
clear
echo
echo-e“\t\t\tSysAdminMenu\n”
echo-e“\t1.Displaydiskspace”
echo-e“\t2.Displayloggedonusers”
echo-e“\t3.Displaymemoryusage”
echo-e“\t0.Exitprogram\n\n”
echo-en“\t\tEnteroption:“
read-n1option
}
while[1]
do
menu
case$optionin
0)
break;;
1)
diskspace;;
2)
whoseon;;
3)
memusage;;
*)
clear
echo“Sorry,wrongselection”;;
esac
echo-en“\n\n\t\t\tHitanykeytocontinue”
read-n1line
done
clear
$
ThismenucreatesthreefunctionstoretrieveadministrativeinformationabouttheLinux
system using common commands. It uses a while loop to continually loop through the
menuuntilthecustomerselectsoption0,whichusesthe breakcommandtobreakoutof
thewhileloop.
You can use this same template to create any shell script menu interface. It provides a
simplewaytointeractwithyourcustomers.
Usingtheselectcommand
Youmayhavenoticedthathalfthechallengeofcreatingatextmenuisjustcreatingthe
menu layout and retrieving the answer that you enter. The bash shell provides a handy
littleutilityforyouthatdoesallthisworkautomatically.
The selectcommandallowsyoutocreateamenufromasinglecommandlineandthen
retrieve the entered answer and automatically process it. The format of the select
commandisasfollows:
selectvariableinlist
do
commands
done
Thelistparameterisaspace-separatedlistoftextitemsthatbuildthemenu.Theselect
commanddisplayseachiteminthelistasanumberedoptionandthendisplaysaspecial
prompt,definedbythePS3environmentvariable,fortheselection.
Here’sasimpleexampleoftheselectcommandinaction:
$catsmenu1
#!/bin/bash
#usingselectinthemenu
functiondiskspace{
clear
df-k
}
functionwhoseon{
clear
who
}
functionmemusage{
clear
cat/proc/meminfo
}
PS3=“Enteroption:“
selectoptionin“Displaydiskspace”“Displayloggedonusers”┐
“Displaymemoryusage”“Exitprogram”
do
case$optionin
“Exitprogram”)
break;;
“Displaydiskspace”)
diskspace;;
“Displayloggedonusers”)
whoseon;;
“Displaymemoryusage”)
memusage;;
*)
clear
echo“Sorry,wrongselection”;;
esac
done
clear
$
The select statement must all be on one line in the code file. That’s indicated by the
continuationcharacterinthelisting.Whenyouruntheprogram,itautomaticallyproduces
thefollowingmenu:
$./smenu1
1)Displaydiskspace3)Displaymemoryusage
2)Displayloggedonusers4)Exitprogram
Enteroption:
Whenyouusetheselectcommand,rememberthattheresultvaluestoredinthevariable
istheentiretextstringandnotthenumberassociatedwiththemenuitem.Thetextstring
valuesarewhatyouneedtocompareinyourcasestatements.
DoingWindows
Usingtextmenusisastepintherightdirection,butthere’sstillsomuchmissinginour
interactivescripts,especiallyifwetrytocomparethemtothegraphicalWindowsworld.
Fortunatelyforus,someveryresourcefulpeopleoutintheopensourceworldhavehelped
usout.
The dialog package is a nifty little tool originally created by Savio Lam and currently
maintainedbyThomasE.Dickey.ThispackagerecreatesstandardWindowsdialogboxes
inatextenvironmentusingANSIescapecontrolcodes.Youcaneasilyincorporatethese
dialogboxesinyourshellscriptstointeractwithyourscriptusers.Thissectiondescribes
thedialogpackageanddemonstrateshowtouseitinshellscripts.
Note
Thedialogpackageisn’tinstalledinallLinuxdistributionsbydefault.Ifit’snot
installedbydefault,becauseofitspopularityit’salmostalwaysincludedinthe
softwarerepository.CheckyourspecificLinuxdistributiondocumentationforhowto
loadthedialogpackage.FortheUbuntuLinuxdistribution,thefollowingisthe
commandlinecommandtoinstallit:
1. sudoapt-getinstalldialog
Thatpackageinstallsthedialogpackageplustherequiredlibrariesforyoursystem.
Thedialogpackage
ThedialogcommandusescommandlineparameterstodeterminewhattypeofWindows
widgettoproduce.AwidgetisthedialogpackagetermforatypeofWindowselement.
ThedialogpackagecurrentlysupportsthetypesofwidgetsshowninTable18.1.
Table18.1ThedialogWidgets
Widget
Description
calendar
Providesacalendarfromwhichtoselectadate
checklist
Displaysmultipleentrieswhereeachentrycanbeturnedonoroff
form
Allowsyoutobuildaformwithlabelsandtextfieldstobefilledout
fselect
Providesafileselectionwindowtobrowseforafile
gauge
Displaysametershowingapercentageofcompletion
infobox
Displaysamessagewithoutwaitingforaresponse
inputbox
Displaysasingletextformboxfortextentry
inputmenu
Providesaneditablemenu
menu
Displaysalistofselectionsfromwhichtochoose
msgbox
DisplaysamessageandrequirestheusertoselectanOKbutton
pause
Displaysametershowingthestatusofaspecifiedpauseperiod
passwordbox
Displaysasingletextboxthathidesenteredtext
passwordform
Displaysaformwithlabelsandhiddentextfields
radiolist Providesagroupofmenuitemswhereonlyoneitemcanbeselected
Displaystextfromafileinascrollwindowusingthetailcommand
tailbox
tailboxbg
textbox
timebox
Sameastailbox,butoperatesinbackgroundmode
Displaysthecontentsofafileinascrollwindow
Providesawindowtoselectanhour,minute,andsecond
yesno
ProvidesasimplemessagewithYesandNobuttons
AsyoucanseefromTable18.1,youcanchoosefromlotsofdifferentwidgets.Thiscan
giveyourscriptsamoreprofessionallookwithverylittleeffort.
Tospecifyaspecificwidgetonthecommandline,youneedtousethedoubledashformat:
dialog—widgetparameters
where widgetisthewidgetnameasseeninTable18.1,and parametersdefinesthesize
ofthewidgetwindowandanytextrequiredforthewidget.
Eachdialogwidgetprovidesoutputintwoforms:
UsingSTDERR
Usingtheexitcodestatus
Theexitcodestatusofthedialogcommanddeterminesthebuttonselectedbytheuser.If
anOKorYesbuttonisselected,the dialogcommandreturnsa0exitstatus.IfaCancel
or No button is selected, the dialog command returns a 1 exit status. You can use the
standard$?variabletodeterminewhichbuttonwasselectedinthedialogwidget.
Ifawidgetreturnsanydata,suchasamenuselection,thedialogcommandsendsthedata
toSTDERR.Youcanusethestandardbashshelltechniqueofredirectingthe STDERRoutput
toanotherfileorfiledescriptor:
dialog—inputbox“Enteryourage:”10202>age.txt
Thiscommandredirectsthetextenteredinthetextboxtotheage.txtfile.
Thefollowingsectionslookatsomeexamplesofthemorecommondialogwidgetsyou’ll
useinyourshellscripts.
Themsgboxwidget
The msgboxwidgetisthemostcommontypeofdialogbox.Itdisplaysasimplemessage
in a window and waits for the user to click an OK button before disappearing. The
followingformatisrequiredtouseamsgboxwidget:
dialog—msgboxtextheightwidth
Thetextparameterisanystringyouwanttoplaceinthewindow.The dialogcommand
automaticallywrapsthetexttofitthesizeofthewindowyoucreate,usingtheheightand
widthparameters.Ifyouwanttoplaceatitleatthetopofthewindow,youcanalsouse
the —title parameter, along with the text of the title. Here’s an example of using the
msgboxwidget:
$dialog—titleTesting—msgbox“Thisisatest”1020
After entering this command, the message box appears on the screen of the terminal
emulatorsessionyou’reusing.Figure18.2showswhatthislookslike.
Figure18.2Usingthemsgboxwidgetinthedialogcommand
If your terminal emulator supports the mouse, you can click the OK button to close the
dialog box. You can also use keyboard commands to simulate a click — just press the
Enterkey.
Theyesnowidget
Theyesnowidgettakesthe msgboxwidgetonestepfurther,allowingtheusertoanswera
yes/no question displayed in the window. It produces two buttons at the bottom of the
window—oneforYesandanotherforNo.Theusercanswitchbetweenbuttonsbyusing
the mouse, the tab key, or the keyboard arrow keys. To select the button, the user can
eitherpressthespacebarortheEnterkey.
Here’sanexampleofusingtheyesnowidget:
$dialog—title“Pleaseanswer”—yesno“Isthisthingon?”1020
$echo$?
1
$
ThisproducesthewidgetshowninFigure18.3.
Figure18.3Usingtheyesnowidgetinthedialogcommand
Theexitstatusofthe dialogcommandissetdependingonwhichbuttontheuserselects.
IftheNobuttonisselected,theexitstatusis1,andiftheYesbuttonisselected,theexit
statusis0.
Theinputboxwidget
Theinputboxwidgetprovidesasimpletextboxareafortheusertoenteratextstring.The
dialog command sends the value of the text string to STDERR. You must redirect that to
retrievetheanswer.Figure18.4demonstrateswhattheinputboxwidgetlookslike.
Figure18.4Theinputboxwidget
AsyoucanseeinFigure18.4,the inputboxprovidestwobuttons—OKandCancel.If
theCancelbuttonisselected,theexitstatusofthecommandis1;otherwise,theexitstatus
is0:
$dialog—inputbox“Enteryourage:”10202>age.txt
$echo$?
0
$catage.txt
12$
You’llnoticewhenyouusethe catcommandtodisplaythecontentsofthetextfilethat
there’s no newline character after the value. This enables you to easily redirect the file
contentstoavariableinashellscripttoextractthestringenteredbytheuser.
Thetextboxwidget
Thetextboxwidgetisagreatwaytodisplaylotsofinformationinawindow.Itproduces
ascrollablewindowcontainingthetextfromafilespecifiedintheparameters:
$dialog—textbox/etc/passwd1545
The contents of the /etc/passwd file are shown within the scrollable text window, as
illustratedinFigure18.5.
Figure18.5Thetextboxwidget
Youcanusethearrowkeystoscrollleftandright,aswellasupanddowninthetextfile.
The bottom line in the window shows the percent location within the file that you’re
viewing.ThetextboxcontainsonlyasingleExitbutton,whichshouldbeselectedtoexit
thewidget.
Themenuwidget
The menu widget allows you to create a window version of the text menu we created
earlierinthischapter.Yousimplyprovideaselectiontagandthetextforeachitem:
$dialog—menu“SysAdminMenu”2030101“Displaydiskspace”
2“Displayusers”3“Displaymemoryusage”4“Exit”2>test.txt
Thefirstparameterdefinesatitleforthemenu.Thenexttwoparametersdefinetheheight
and width of the menu window, while the third parameter defines the number of menu
itemsthatappearinthewindowatonetime.Iftherearemoremenuitems,youcanscroll
throughthemusingthearrowkeys.
Following those parameters, you must add menu item pairs. The first element is the tag
usedtoselectthemenuitem.Eachtagshouldbeuniqueforeachmenuitemandcanbe
selectedbypressingtheappropriatekeyonthekeyboard.Thesecondelementisthetext
usedinthemenu.Figure18.6demonstratesthemenuproducedbytheexamplecommand.
Figure18.6Themenuwidgetwithmenuitems
Iftheuserselectsamenuitembypressingtheappropriatekeyforthetag,thatmenuitem
ishighlightedbutnotselected.Aselectionisn’tmadeuntiltheOKbuttonisselectedby
usingeitherthemouseortheEnterkey.The dialog command sends the selected menu
itemtexttoSTDERR,whichyoucanredirectasneeded.
Thefselectwidget
There are several fancy built-in widgets provided by the dialog command. The fselect
widget is extremely handy when working with filenames. Instead of forcing the user to
typeafilename,youcanusethe fselectwidgettobrowsetothefilelocationandselect
thefile,asshowninFigure18.7.
Figure18.7Thefselectwidget
Thefselectwidgetformatlookslike:
$dialog—title“Selectafile”—fselect$HOME/10502>file.txt
The first parameter after the fselect option is the starting folder location used in the
window.Thefselectwidgetwindowconsistsofadirectorylistingontheleftside,afile
listing on the right side that shows all the files in the selected directory, and a simple
textbox that contains the currently selected file or directory. You can manually type a
filenameinthetextbox,oryoucanusethedirectoryandfilelistingstoselectone(usethe
spacebartoselectafiletoaddtothetextbox).
Thedialogoptions
In addition to the standard widgets, you can customize lots of different options in the
dialogcommand.You’vealreadyseenthe —titleparameterinaction.Thisallowsyouto
setatitleforthewidgetthatappearsatthetopofthewindow.
Lots of other options allow you to completely customize both the appearance and the
behavior of your windows. Table 18.2 shows the options available for the dialog
command.
Table18.2ThedialogCommandOptions
Option
—aspectratio
Description
ProceedstothenextdialogunlessEscortheCancelbuttonhasbeen
pressed
Specifiesthewidth/heightaspectratioofthewindow
—backtitle
title
Specifiesatitletodisplayonthebackground,atthetopofthescreen
—add-widget
—beginxy
Specifiesthestartinglocationofthetop-leftcornerofthewindow
—cancel-label
label
SpecifiesanalternativelabelfortheCancelbutton
Option
—defaultno
Description
Clearsthedisplayusingthedefaultdialogbackgroundcolor
EmbedsANSIcolorcodesindialogtext
Allowsnewlinecharactersindialogtextandforcesalinewrap
Dumpsasampleconfigurationfiletothespecifiedfile
Makesthedefaultofayes/nodialogNo
—default-item
string
Setsthedefaultiteminachecklist,form,ormenudialog
—exit-label
label
SpecifiesanalternativelabelfortheExitbutton
—extra-button
DisplaysanextrabuttonbetweentheOKandCancelbuttons
—extra-label
label
SpecifiesanalternativelabelfortheExtrabutton
—help
—help-button
Displaysthedialogcommandhelpmessage
DisplaysaHelpbuttonaftertheOKandCancelbuttons
—help-label
label
SpecifiesanalternativelabelfortheHelpbutton
—clear
—colors
—cr-wrap
—create-rcfile
Writesthechecklist,radiolist,orforminformationafterthehelp
informationintheHelpbuttonwasselected
—ignore
Ignoresoptionsthatdialogdoesnotrecognize
—input-fdfd
Specifiesanalternativefiledescriptor,otherthanSTDIN
—insecure
Changesthepasswordwidgettodisplayasteriskswhentyping
Addsahelpcolumnatthebottomofthescreenforeachtagina
—item-help
checklist,radiolist,ormenuforthetagitem
—keep-window
Doesn’tclearoldwidgetsfromthescreen
—max-inputsize
Specifiesamaximumstringsizefortheinput;defaultis2048
—nocancel
SuppressestheCancelbutton
—no-collapse
Doesn’tconverttabstospacesindialogtext
PlacesthetailboxbgdialoginbackgroundanddisablesSIGHUPfor
—no-kill
theprocess
—no-labellabel
SpecifiesanalternativelabelfortheNobutton
—no-shadow
Doesn’tdisplayshadowsfordialogwindows
—ok-labellabel
SpecifiesanalternativelabelfortheOKbutton
—help-status
—output-fdfd
—print-maxsize
—print-size
—print-version
SpecifiesanalternativeoutputfiledescriptorotherthanSTDERR
Printsthemaximumsizeofdialogwindowsallowedtotheoutput
Printsthesizeofeachdialogwindowtotheoutput
Printsthedialogversiontooutput
—separateoutput
Outputstheresultofachecklistwidgetonelineatatimewithno
quoting
—separator
string
Specifiesastringthatseparatestheoutputforeachwidget
—separatewidgetstring
Specifiesastringthatseparatestheoutputforeachwidget
Drawsashadowtotherightandbottomofeachwindow
—single-quoted
Usessinglequotingifneededforthechecklistoutput
Delaysforthespecifiednumberofsecondsafterprocessingthedialog
—sleepsec
window
—stderr
SendsoutputtoSTDERR—thedefaultbehavior
—stdout
SendsoutputtoSTDOUT
—tab-correct
Convertstabstospaces
—tab-lenn
Specifiesthenumberofspacesatabcharacteruses;defaultis8
Specifiesthenumberofsecondsbeforeexitingwithanerrorcodeifno
—timeoutsec
userinput
—titletitle
Specifiesthetitleofthedialogwindow
—trim
Removesleadingspacesandnewlinecharactersfromdialogtext
—visit-items Modifiesthetabstopsinthedialogwindowtoincludethelistofitems
—shadow
—yes-label
label
SpecifiesanalternativelabelfortheYesbutton
The—backtitleoptionisahandywaytocreateacommontitleforyourmenuthroughthe
script. If you specify it for each dialog window, it persists throughout your application,
creatingaprofessionallooktoyourscript.
AsyoucantellfromTable18.2,youcanoverwriteanyofthebuttonlabelsinyourdialog
window.Thisfeatureallowsyoutocreatejustaboutanywindowsituationyouneed.
Usingthedialogcommandinascript
Usingthe dialogcommandinyourscriptsisasnap.Therearejusttwothingsyoumust
remember:
Checktheexitstatusofthedialogcommandifthere’saCancelorNobutton
available.
RedirectSTDERRtoretrievetheoutputvalue.
If you follow these two rules, you’ll have a professional-looking interactive script in no
time. Here’s an example using dialog widgets to reproduce the system admin menu
createdearlierinthechapter:
$catmenu3
#!/bin/bash
#usingdialogtocreateamenu
temp=$(mktemp-ttest.XXXXXX)
temp2=$(mktemp-ttest2.XXXXXX)
functiondiskspace{
df-k>$temp
dialog—textbox$temp2060
}
functionwhoseon{
who>$temp
dialog—textbox$temp2050
}
functionmemusage{
cat/proc/meminfo>$temp
dialog—textbox$temp2050
}
while[1]
do
dialog—menu“SysAdminMenu”2030101“Displaydiskspace”2
“Displayusers”3“Displaymemoryusage”0“Exit”2>$temp2
if[$?-eq1]
then
break
fi
selection=$(cat$temp2)
case$selectionin
1)
diskspace;;
2)
whoseon;;
3)
memusage;;
0)
break;;
*)
dialog—msgbox“Sorry,invalidselection”1030
esac
done
rm-f$temp2>/dev/null
rm-f$temp22>/dev/null
$
The script uses the while loop with a constant true value to create an endless loop
displaying the menu dialog. This means that, after every function, the script returns to
displayingthemenu.
ThemenudialogincludesaCancelbutton,sothescriptcheckstheexitstatusofthedialog
commandincasetheuserpressestheCancelbuttontoexit.Becauseit’sina whileloop,
exitingisaseasyasusingthebreakcommandtojumpoutofthewhileloop.
Thescriptusesthemktempcommandtocreatetwotemporaryfilesforholdingdataforthe
dialogcommands.Thefirstone,$temp,isusedtoholdtheoutputofthedf,whoeson,and
meminfocommandssotheycanbedisplayedinthe textboxdialog(seeFigure18.8).The
second temporary file, $temp2, is used to hold the selection value from the main menu
dialog.
Figure18.8Thememinfocommandoutputdisplayedusingthetextboxdialogoption
Nowthisisstartingtolooklikearealapplicationthatyoucanshowofftopeople!
GettingGraphic
Ifyou’relookingforevenmoregraphicsforyourinteractivescripts,youcangoonestep
further.BoththeKDEandGNOMEdesktopenvironments(seeChapter1)haveexpanded
onthe dialog command idea and include commands that produce X Window graphical
widgetsfortheirrespectiveenvironments.
Thissectiondescribesthekdialogandzenitypackages,whichprovidegraphicalwindow
widgetsfortheKDEandGNOMEdesktops,respectively.
TheKDEenvironment
The KDE graphical environment includes the kdialog package by default. The kdialog
packageusesthe kdialogcommandtogeneratestandardwindows,similartothedialogstyle widgets, within your KDE desktop. However, instead of having the clunky feel to
them,thesewindowsblendrightinwiththerestofyourKDEapplicationwindows!This
allowsyoutoproduceWindows-qualityuserinterfacesdirectlyfromyourshellscripts!
Note
JustbecauseyourLinuxdistributionusestheKDEdesktopdoesn’tnecessarilymean
ithasthekdialogpackageinstalledbydefault.Youmayneedtomanuallyinstallit
fromthedistributionrepository.
kdialogwidgets
Just like the dialog command, the kdialog command uses command line options to
specifywhattypeofwindowwidgettouse.Thefollowingistheformatofthe kdialog
command:
kdialogdisplay-optionswindow-optionsarguments
The window-options options allow you to specify what type of window widget to use.
TheavailableoptionsareshowninTable18.3.
Table18.3kdialogWindowOptions
Option
—warningyesnotext
Description
Achecklistmenu,withstatusspecifyingiftheitemis
checkedornot
Errormessagebox
Inputtextboxwhereyoucanspecifythedefaultvalueusing
theinitvalue
Menuselectionboxtitleandalistofitemsidentifiedbya
tag
Simplemessageboxwithspecifiedtext
Passwordinputtextboxthathidesuserinput
Aradiolistmenu,withstatusspecifyingiftheitemis
selectedornot
Returnsitemsonseparatelinesforchecklistandradiolist
menus
Sorrymessagebox
Textboxdisplayingthecontentsoffile,alternatively
specifiedbywidthandheight
SpecifiesatitlefortheTitleBarareaofthedialogwindow
WarningmessageboxwithYesandNobuttons
—warningcontinuecancel
text
WarningmessageboxwithContinueandCancelbuttons
—warningyesnocanceltext
WarningmessageboxwithYes,No,andCancelbuttons
QuestionboxwithYesandNobuttons
QuestionboxwithYes,No,andCancelbuttons
—checklisttitle[tag
itemstatus]
—errortext
—inputboxtext[init]
—menutitle[tagitem]
—msgboxtext
—passwordtext
—radiolisttitle[tag
itemstatus]
—separate-output
—sorrytext
—textboxfile[width]
[height]
—titletitle
—yesnotext
—yesnocanceltext
AsyoucanseefromTable18.3,allthestandardwindowdialogboxtypesarerepresented.
However,whenyouuseakdialogwindowwidget,itappearsasaseparatewindowinthe
KDEdesktop,notinsidetheterminalemulatorsession!
The checklistand radiolist widgets allow you to define individual items in the lists
andwhethertheyareselectedbydefault:
$kdialog—checklist“ItemsIneed”1“Toothbrush”on2“Toothpaste”
off3“Hairbrush”on4“Deodorant”off5“Slippers”off
TheresultingchecklistwindowisshowninFigure18.9.
Figure18.9Akdialogchecklistdialogwindow
Theitemsspecifiedas“on”arehighlightedinthechecklist.Toselectordeselectanitem
inthechecklist,justclickit.IfyouselecttheOKbutton,thekdialogsendsthetagvalues
toSTDOUT:
“1”“3”
$
When you press the Enter key, the kdialog box appears with the selections. When you
clicktheOKorCancelbuttons,thekdialogcommandreturnseachtagasastringvalueto
STDOUT(thesearethe”1”,and ”3”valuesyouseeintheoutput).Yourscriptmustbeable
toparsetheresultingvaluesandmatchthemwiththeoriginalvalues.
Usingkdialog
Youcanusethe kdialogwindowwidgetsinyourshellscriptssimilarlytohowyouuse
thedialogwidgets.Thebigdifferenceisthatthe kdialogwindowwidgetsoutputvalues
usingSTDOUTinsteadofSTDERR.
Here’sascriptthatconvertsthesysadminmenucreatedearlierintoaKDEapplication:
$catmenu4
#!/bin/bash
#usingkdialogtocreateamenu
temp=$(mktemp-ttemp.XXXXXX)
temp2=$(mktemp-ttemp2.XXXXXX)
functiondiskspace{
df-k>$temp
kdialog—textbox$temp100010
}
functionwhoseon{
who>$temp
kdialog—textbox$temp50010
}
functionmemusage{
cat/proc/meminfo>$temp
kdialog—textbox$temp300500
}
while[1]
do
kdialog—menu“SysAdminMenu”“1”“Displaydiskspace”“2”“Display
users”“3”“Displaymemoryusage”“0”“Exit”>$temp2
if[$?-eq1]
then
break
fi
selection=$(cat$temp2)
case$selectionin
1)
diskspace;;
2)
whoseon;;
3)
memusage;;
0)
break;;
*)
kdialog—msgbox“Sorry,invalidselection”
esac
done
$
Thereisn’tmuchdifferenceinthescriptfromusingthekdialogcommandandthedialog
command.TheresultingmainmenugeneratedisshowninFigure18.10.
Figure18.10Thesysadminmenuscriptusingkdialog
NowyoursimpleshellscriptlooksjustlikearealKDEapplication!There’snolimitto
whatyoucandowithyourinteractivescriptsnow.
TheGNOMEenvironment
The GNOME graphical environment supports two popular packages that can generate
standardwindows:
gdialog
zenity
By far, zenity is the most commonly available package found in most GNOME desktop
Linux distributions (it’s installed by default in both Ubuntu and Fedora). This section
describesthefeaturesofzenityanddemonstrateshowtouseitinyourshellscripts.
zenityWidgets
As you would expect, zenity allows you to create different windows widgets by using
commandlineoptions.Table18.4showsthedifferentwidgetsthatzenitycanproduce.
Table18.4ThezenityWindowsWidgets
Option
Description
—calendar
Displaysafullmonthcalendar
—entry
Displaysatextentrydialogwindow
—error
Displaysanerrormessagedialogwindow
—file-selection Displaysafullpathnameandfilenamedialogwindow
—info
Displaysaninformationaldialogwindow
—list
Displaysachecklistorradiolistdialogwindow
—notification
Displaysanotificationicon
—progress
Displaysaprogressbardialogwindow
—question
—scale
—text-info
—warning
Displaysayes/noquestiondialogwindow
Displaysascaledialogwindow
Displaysatextboxcontainingtext
Displaysawarningdialogwindow
The zenity command line program works somewhat differently than the kdialog and
dialog programs. Many of the widget types are defined using additional options on the
commandline,insteadofincludingthemasargumentstoanoption.
The zenity command does offer some pretty cool advanced dialog windows. The
calendaroptionproducesafullmonthcalendar,asshowninFigure18.11.
Figure18.11Thezenitycalendardialogwindow
When you select a date from the calendar, the zenity command returns the value to
STDOUT,justlikekdialog:
$zenity—calendar
12/25/2011
$
Anotherprettycoolwindowinzenityisthefileselectionoption,showninFigure18.12.
Figure18.12Thezenityfileselectiondialogwindow
Youcanusethedialogwindowtobrowsetoanydirectorylocationonthesystem(aslong
asyouhavetheprivilegestoviewthedirectory)andselectafile.Whenyouselectafile,
thezenitycommandreturnsthefullfileandpathname:
$zenity—file-selection
/home/ubuntu/menu5
$
Withtoolslikethatatyourdisposal,thesky’sthelimitwithyourshellscriptcreations!
Usingzenityinscripts
Asyouwouldexpect,zenityperformswellinshellscripts.Unfortunately,zenitychosenot
tofollowtheoptionconventionusedin dialogand kdialog,soconvertinganyexisting
interactivescriptstozenitymayprovechallenging.
In converting the sys admin menu from kdialogto zenity, we had to do quite a bit of
manipulationofthewidgetdefinitions:
$catmenu5
#!/bin/bash
#usingzenitytocreateamenu
temp=$(mktemp-ttemp.XXXXXX)
temp2=$(mktemp-ttemp2.XXXXXX)
functiondiskspace{
df-k>$temp
zenity—text-info—title“Diskspace”—filename=$temp
—width750—height10
}
functionwhoseon{
who>$temp
zenity—text-info—title“Loggedinusers”—filename=$temp
—width500—height10
}
functionmemusage{
cat/proc/meminfo>$temp
zenity—text-info—title“Memoryusage”—filename=$temp
—width300—height500
}
while[1]
do
zenity—list—radiolist—title“SysAdminMenu”—column“Select”
—column“MenuItem”FALSE“Displaydiskspace”FALSE“Displayusers”
FALSE“Displaymemoryusage”FALSE“Exit”>$temp2
if[$?-eq1]
then
break
fi
selection=$(cat$temp2)
case$selectionin
“Displaydiskspace”)
diskspace;;
“Displayusers”)
whoseon;;
“Displaymemoryusage”)
memusage;;
Exit)
break;;
*)
zenity—info“Sorry,invalidselection”
esac
done
$
Becausezenitydoesn’tsupportthemenudialogwindow,weusedaradiolisttypewindow
forthemainmenu,asshowninFigure18.13.
Figure18.13Thesysadminmenuusingzenity
The radiolist uses two columns, each with a column heading. The first column includes
theradiobuttonstoselect.Thesecondcolumnistheitemtext.Theradiolistalsodoesn’t
use tags for the items. When you select an item, the full text of the item is returned to
STDOUT.Thismakeslifealittlemoreinterestingforthe casecommand.Youmustusethe
fulltextfromtheitemsinthecaseoptions.Ifthereareanyspacesinthetext,youneedto
usequotationmarksaroundthetext.
Usingthezenitypackage,youcanaddaWindowsfeeltoyourinteractiveshellscriptsin
theGNOMEdesktop.
Summary
Interactiveshellscriptshaveareputationforbeingdullandboring.Youcanchangethat
byusingafewdifferenttechniquesandtoolsavailableonmostLinuxsystems.First,you
cancreatemenusystemsforyourinteractivescriptsbyusingthecasecommandandshell
scriptfunctions.
The menucommandallowsyoutopaintamenu,usingthestandard echocommand,and
readaresponsefromtheuser,usingthe readcommand.The casecommandthenselects
theappropriateshellscriptfunctionbasedonthevalueentered.
The dialog program provides several prebuilt text widgets for creating Windows-like
objectsonatext-basedterminalemulator.Youcancreatedialogboxesfordisplayingtext,
enteringtext,andchoosingfilesanddatesbyusingthedialogprogram.Thishelpsbring
evenmorelifetoyourshellscript.
Ifyou’rerunningyourshellscriptsinagraphicalXWindowenvironment,youcanutilize
even more tools in your interactive scripts. For the KDE desktop, there’s the kdialog
program.Thisprogramprovidessimplecommandstocreatewindowswidgetsforallthe
basic windows functions. For the GNOME desktop, there are the gdialog and zenity
programs.EachoftheseprogramsprovideswindowwidgetsthatblendintotheGNOME
desktopjustlikearealWindowsapplication.
Thenextchapterdivesintothesubjectofeditingandmanipulatingtextdatafiles.Often
the biggest use of shell scripts revolves around parsing and displaying data in text files
suchasloganderrorfiles.TheLinuxenvironmentincludestwoveryusefultools,sedand
gawk,forworkingwithtextdatainyourshellscripts.Thenextchapterintroducesyouto
thesetools,andshowsthebasicsofhowtousethem.
Chapter19
Introducingsedandgawk
InThisChapter
1. LearningaboutthesedEditor
2. GettingintroducedtothegawkEditor
3. ExploringsedEditorbasics
Byfar,oneofthemostcommonfunctionsthatpeopleuseshellscriptsforistowork
withtextfiles.Betweenexamininglogfiles,readingconfigurationfiles,andhandling
dataelements,shellscriptscanhelpautomatethemundanetasksofmanipulatingany
typeofdatacontainedintextfiles.However,tryingtomanipulatethecontentsoftext
filesusingjustshellscriptcommandscanbesomewhatawkward.Ifyouperformany
typeofdatamanipulationinyourshellscripts,youwanttobecomefamiliarwiththe
sedandgawktoolsavailableinLinux.Thesetoolscangreatlysimplifyanydatahandlingtasksyouneedtoperform.
ManipulatingText
Chapter10showedyouhowtoedittextfilesusingdifferenteditorprogramsavailablein
theLinuxenvironment.Theseeditorsenableyoutoeasilymanipulatetextcontainedina
textfilebyusingsimplecommandsormouseclicks.
Therearetimes,however,whenyou’llfindyourselfwantingtomanipulatetextinatext
file on the fly, without having to pull out a full-fledged interactive text editor. In these
situations, it would be useful to have a simple command line editor that could easily
format,insert,modify,ordeletetextelementsautomatically.
TheLinuxsystemprovidestwocommontoolsfordoingjustthat.Thissectiondescribes
thetwomostpopularcommandlineeditorsusedintheLinuxworld,sedandgawk.
Gettingtoknowthesededitor
The sededitoriscalledastreameditor,asopposedtoanormalinteractivetexteditor.In
aninteractivetexteditor,suchasvim,youinteractivelyusekeyboardcommandstoinsert,
delete,orreplacetextinthedata.Astreameditoreditsastreamofdatabasedonasetof
rulesyousupplyaheadoftime,beforetheeditorprocessesthedata.
The sededitorcanmanipulatedatainadatastreambasedoncommandsyoueitherenter
intothecommandlineorstoreinacommandtextfile.Thesededitordoesthesethings:
1. Readsonedatalineatatimefromtheinput
2. Matchesthatdatawiththesuppliededitorcommands
3. Changesdatainthestreamasspecifiedinthecommands
4. OutputsthenewdatatoSTDOUT
Afterthestreameditormatchesallthecommandsagainstalineofdata,itreadsthenext
lineofdataandrepeatstheprocess.Afterthestreameditorprocessesallthelinesofdata
inthestream,itterminates.
Because the commands are applied sequentially line by line, the sed editor makes only
onepassthroughthedatastreamtomaketheedits.Thismakesthesededitormuchfaster
thananinteractiveeditorandallowsyoutoquicklymakechangestodatainafileonthe
fly.
Here’stheformatforusingthesedcommand:
sedoptionsscriptfile
The options parameters allow you to customize the behavior of the sed command and
includetheoptionsshowninTable19.1.
Table19.1ThesedCommandOptions
Option
-e
Description
Addscommandsspecifiedinthescripttothecommandsrunwhileprocessing
theinput
Addsthecommandsspecifiedinthefiletothecommandsrunwhile
-ffile
processingtheinput
-n
Doesn’tproduceoutputforeachcommand,butwaitsfortheprintcommand
script
Thescriptparameterspecifiesasinglecommandtoapplyagainstthestreamdata.Ifmore
thanonecommandisrequired,youmustuseeitherthe -eoptiontospecifytheminthe
commandlineorthe-foptiontospecifytheminaseparatefile.Numerouscommandsare
availableformanipulatingdata.Weexaminesomeofthebasiccommandsusedbythesed
editorinthischapterandthenlookatsomeofthemoreadvancedcommandsinChapter
21.
Defininganeditorcommandinthecommandline
Bydefault,the sededitorappliesthespecifiedcommandstothe STDINinputstream.This
allowsyoutopipedatadirectlytothe sededitorforprocessing.Here’saquickexample
demonstratinghowtodothis:
$echo“Thisisatest”|sed‘s/test/bigtest/’
Thisisabigtest
$
Thisexampleusesthe scommandinthe sededitor.The scommandsubstitutesasecond
text string for the first text string pattern specified between the forward slashes. In this
example,thewordsbigtestweresubstitutedforthewordtest.
Whenyourunthisexample,itshoulddisplaytheresultsalmostinstantaneously.That’sthe
powerofusingthesededitor.Youcanmakemultipleeditstodatainaboutthesametime
ittakesforsomeoftheinteractiveeditorsjusttostartup.
Of course, this simple test just edited one line of data. You should get the same speedy
resultswheneditingcompletefilesofdata:
$catdata1.txt
Thequickbrownfoxjumpsoverthelazydog.
Thequickbrownfoxjumpsoverthelazydog.
Thequickbrownfoxjumpsoverthelazydog.
Thequickbrownfoxjumpsoverthelazydog.
$
$sed‘s/dog/cat/’data1.txt
Thequickbrownfoxjumpsoverthelazycat.
Thequickbrownfoxjumpsoverthelazycat.
Thequickbrownfoxjumpsoverthelazycat.
Thequickbrownfoxjumpsoverthelazycat.
$
The sed command executes and returns the data almost instantaneously. As it processes
eachlineofdata,theresultsaredisplayed.You’llstartseeingresultsbeforethesededitor
completesprocessingtheentirefile.
It’simportanttonotethatthe sededitordoesn’tmodifythedatainthetextfileitself.It
only sends the modified text to STDOUT. If you look at the text file, it still contains the
originaldata:
$catdata1.txt
Thequickbrownfoxjumpsoverthelazydog.
Thequickbrownfoxjumpsoverthelazydog.
Thequickbrownfoxjumpsoverthelazydog.
Thequickbrownfoxjumpsoverthelazydog.
$
Usingmultipleeditorcommandsinthecommandline
Toexecutemorethanonecommandfromthesedcommandline,justusethe-eoption:
$sed-e‘s/brown/green/;s/dog/cat/’data1.txt
Thequickgreenfoxjumpsoverthelazycat.
Thequickgreenfoxjumpsoverthelazycat.
Thequickgreenfoxjumpsoverthelazycat.
Thequickgreenfoxjumpsoverthelazycat.
$
Both commands are applied to each line of data in the file. The commands must be
separated with a semicolon, and there shouldn’t be any spaces between the end of the
commandandthesemicolon.
Insteadofusingasemicolontoseparatethecommands,youcanusethesecondaryprompt
inthebashshell.Justenterthefirstsinglequotationmarktoopenthe sedprogramscript
(sed editor command list), and bash continues to prompt you for more commands until
youentertheclosingquotationmark:
$sed-e‘
>s/brown/green/
>s/fox/elephant/
>s/dog/cat/’data1.txt
Thequickgreenelephantjumpsoverthelazycat.
Thequickgreenelephantjumpsoverthelazycat.
Thequickgreenelephantjumpsoverthelazycat.
Thequickgreenelephantjumpsoverthelazycat.
$
You must remember to finish the command on the same line where the closing single
quotation mark appears. After the bash shell detects the closing quotation mark, it
processes the command. After it starts, the sed command applies each command you
specifiedtoeachlineofdatainthetextfile.
Readingeditorcommandsfromafile
Finally,ifyouhavelotsof sedcommandsyouwanttoprocess,itisofteneasiertojust
storetheminaseparatefile.Usethe-foptiontospecifythefileinthesedcommand:
$catscript1.sed
s/brown/green/
s/fox/elephant/
s/dog/cat/
$
$sed-fscript1.seddata1.txt
Thequickgreenelephantjumpsoverthelazycat.
Thequickgreenelephantjumpsoverthelazycat.
Thequickgreenelephantjumpsoverthelazycat.
Thequickgreenelephantjumpsoverthelazycat.
$
In this case, you don’t put a semicolon after each command. The sededitorknowsthat
eachlinecontainsaseparatecommand.Aswithenteringcommandsonthecommandline,
thesededitorreadsthecommandsfromthespecifiedfileandappliesthemtoeachlinein
thedatafile.
Tip
Itcanbeeasytoconfuseyoursededitorscriptfileswithyourbashshellscriptfiles.
Toeliminateconfusion,usea.sedfileextensiononyoursedscriptfiles.
We’lllookatsomeother sededitorcommandsthatcomeinhandyformanipulatingdata
inthe“Commandingatthe sedEditorBasics”section.Beforethat,let’squicklylookat
theotherLinuxdataeditor.
Gettingtoknowthegawkprogram
Although the sed editor is a handy tool for modifying text files on the fly, it has its
limitations.Often,youneedamoreadvancedtoolformanipulatingdatainafile,onethat
provides a more programming-like environment allowing you to modify and reorganize
datainafile.Thisiswheregawkcomesin.
Note
Thegawkprogramisnotinstalledbydefaultonalldistributions.IfyourLinux
distributiondoesnothavethegawkprogram,installthegawkpackageusingChapter9
asaguide.
The gawk program is the GNU version of the original awk program in Unix. The gawk
program takes stream editing one step further than the sed editor by providing a
programming language instead of just editor commands. Within the gawk programming
language,youcandothefollowing:
Definevariablestostoredata.
Usearithmeticandstringoperatorstooperateondata.
Usestructuredprogrammingconcepts,suchasif-thenstatementsandloops,toadd
logictoyourdataprocessing.
Generateformattedreportsbyextractingdataelementswithinthedatafileand
repositioningtheminanotherorderorformat.
The gawk program’s report-generating capabilities are often used for extracting data
elements from large bulky text files and formatting them into a readable report. The
perfectexampleofthisisformattinglogfiles.Tryingtoporethroughlinesoferrorsina
logfilecanbedifficult.The gawkprogramallowsyoutofilterjustthedataelementsyou
want to view from the log file, and then you can format them in a manner that makes
readingtheimportantdataeasier.
Visitingthegawkcommandformat
Here’sthebasicformatofthegawkprogram:
gawkoptionsprogramfile
Table19.2showstheoptionsavailablewiththegawkprogram.
Table19.2ThegawkOptions
Option
Description
-Ffs
Specifiesafileseparatorfordelineatingdatafieldsinaline
-ffile
Specifiesafilenametoreadtheprogramfrom
-vvar=value
Definesavariableanddefaultvalueusedinthegawkprogram
-mfN
Specifiesthemaximumnumberoffieldstoprocessinthedatafile
-mrN
Specifiesthemaximumrecordsizeinthedatafile
-Wkeyword
Specifiesthecompatibilitymodeorwarninglevelforgawk
The command line options provide an easy way to customize features in the gawk
program.We’lllookmorecloselyattheseasweexploregawk.
Thepowerofgawkisintheprogramscript.Youcanwritescriptstoreadthedatawithina
textlineandthenmanipulateanddisplaythedatatocreateanytypeofoutputreport.
Readingtheprogramscriptfromthecommandline
A gawk program script is defined by opening and closing braces. You must place script
commandsbetweenthetwobraces({}).Ifyouincorrectlyuseaparenthesisinsteadofa
bracetoencloseyourgawkscript,yougeterrormessages,similartothefollowing:
$gawk‘(print“HelloWorld!”}’
gawk:(print“HelloWorld!”}
gawk:∧syntaxerror
Becausethe gawkcommandlineassumesthatthescriptisasingletextstring,youmust
also enclose your script in single quotation marks. Here’s an example of a simple gawk
programscriptspecifiedonthecommandline:
$gawk‘{print“HelloWorld!”}’
Theprogramscriptdefinesasinglecommand,the printcommand.The printcommand
does what it says: It prints text to STDOUT. If you try running this command, you’ll be
somewhat disappointed, because nothing happens right away. Because no filename was
definedinthecommandline,the gawkprogramretrievesdatafrom STDIN.Whenyourun
theprogram,itjustwaitsfortexttocomeinviaSTDIN.
IfyoutypealineoftextandpresstheEnterkey, gawkrunsthetextthroughtheprogram
script.Justlikethe sededitor,the gawkprogramexecutestheprogramscriptoneachline
of text available in the data stream. Because the program script is set to display a fixed
textstring,nomatterwhattextyouenterinthedatastream,yougetthesametextoutput:
$gawk‘{print“HelloWorld!”}’
Thisisatest
HelloWorld!
hello
HelloWorld!
Thisisanothertest
HelloWorld!
Toterminatethe gawkprogram,youmustsignalthatthedatastreamhasended.Thebash
shellprovidesakeycombinationtogenerateanEnd-of-File(EOF)character.TheCtrl+D
key combination generates an EOF character in bash. Using that key combination
terminatesthegawkprogramandreturnsyoutoacommandlineinterfaceprompt.
Usingdatafieldvariables
Oneoftheprimaryfeaturesofgawkisitsabilitytomanipulatedatainthetextfile.Itdoes
thisbyautomaticallyassigningavariabletoeachdataelementinaline.Bydefault, gawk
assignsthefollowingvariablestoeachdatafielditdetectsinthelineoftext:
$0representstheentirelineoftext.
$1representsthefirstdatafieldinthelineoftext.
$2representstheseconddatafieldinthelineoftext.
$nrepresentsthenthdatafieldinthelineoftext.
Each data field is determined in a text line by a fieldseparationcharacter. When gawk
reads a line of text, it delineates each data field using the defined field separation
character.Thedefaultfieldseparationcharacterin gawkisanywhitespacecharacter(such
asthetaborspacecharacters).
Here’sanexamplegawkprogramthatreadsatextfileanddisplaysonlythefirstdatafield
value:
$catdata2.txt
Onelineoftesttext.
Twolinesoftesttext.
Threelinesoftesttext.
$
$gawk‘{print$1}’data2.txt
One
Two
Three
$
Thisprogramusesthe $1fieldvariabletodisplayonlythefirstdatafieldforeachlineof
text.
Ifyou’rereadingafilethatusesadifferentfieldseparationcharacter,youcanspecifyitby
usingthe-Foption:
$gawk-F:‘{print$1}’/etc/passwd
root
bin
daemon
adm
lp
sync
shutdown
halt
mail
[…]
Thisshortprogramdisplaysthefirstdatafieldinthepasswordfileonthesystem.Because
the/etc/passwdfileusesacolontoseparatethedatafields,ifyouwanttoseparateeach
dataelement,youmustspecifyitasthefieldseparationcharacterinthegawkoptions.
Usingmultiplecommandsintheprogramscript
A programming language wouldn’t be very useful if you could only execute one
command. The gawk programming language allows you to combine commands into a
normal program. To use multiple commands in the program script specified on the
commandline,justplaceasemicolonbetweeneachcommand:
$echo“MynameisRich”|gawk‘{$4=“Christine”;print$0}’
MynameisChristine
$
The first command assigns a value to the $4 field variable. The second command then
prints the entire data field. Notice from the output that the gawk program replaced the
fourthdatafieldintheoriginaltextwiththenewvalue.
Youcanalsousethesecondaryprompttoenteryourprogramscriptcommandsonelineat
atime:
$gawk‘{
>$4=“Christine”
>print$0}’
MynameisRich
MynameisChristine
$
Afteryouopenthesinglequotationmark,thebashshellprovidesthesecondarypromptto
promptyouformoredata.Youcanaddyourcommandsoneatatimeoneachlineuntil
you enter the closing single quotation mark. Because no filename was defined in the
commandline,thegawkprogramretrievesdatafromSTDIN.Whenyouruntheprogram,it
waits for text to come in via STDIN. To exit the program, just press the Ctrl+D key
combinationtosignaltheendofthedata.
Readingtheprogramfromafile
As with the sed editor, the gawk editor allows you to store your programs in a file and
refertotheminthecommandline:
$catscript2.gawk
{print$1“‘shomedirectoryis”$6}
$
$gawk-F:-fscript2.gawk/etc/passwd
root’shomedirectoryis/root
bin’shomedirectoryis/bin
daemon’shomedirectoryis/sbin
adm’shomedirectoryis/var/adm
lp’shomedirectoryis/var/spool/lpd
[…]
Christine’shomedirectoryis/home/Christine
Samantha’shomedirectoryis/home/Samantha
Timothy’shomedirectoryis/home/Timothy
$
The script2.gawk program script uses the print command again to print the
/etc/passwdfile’shomedirectorydatafield(fieldvariable $6)andtheuseriddatafield
(fieldvariable$1).
You can specify multiple commands in the program file. To do so, just place each
commandonaseparateline.Youdon’tneedtousesemicolons:
$catscript3.gawk
{
text=“‘shomedirectoryis“
print$1text$6
}
$
$gawk-F:-fscript3.gawk/etc/passwd
root’shomedirectoryis/root
bin’shomedirectoryis/bin
daemon’shomedirectoryis/sbin
adm’shomedirectoryis/var/adm
lp’shomedirectoryis/var/spool/lpd
[…]
Christine’shomedirectoryis/home/Christine
Samantha’shomedirectoryis/home/Samantha
Timothy’shomedirectoryis/home/Timothy
$
Thescript3.gawkprogramscriptdefinesavariabletoholdatextstringusedintheprint
command.Noticethatgawkprogramsdon’tuseadollarsignwhenreferencingavariable’s
value,asashellscriptdoes.
Runningscriptsbeforeprocessingdata
The gawkprogramalsoallowsyoutospecifywhentheprogramscriptisrun.Bydefault,
gawkreadsalineoftextfromtheinputandthenexecutestheprogramscriptonthedatain
thelineoftext.Sometimes,youmayneedtorunascriptbeforeprocessingdata,suchasto
create a header section for a report. The BEGIN keyword is used to accomplish this. It
forces gawktoexecutetheprogramscriptspecifiedafterthe BEGINkeyword,before gawk
readsthedata:
$gawk‘BEGIN{print“HelloWorld!”}’
HelloWorld!
$
Thistimethe printcommanddisplaysthetextbeforereadinganydata.However,afterit
displaysthetext,itquicklyexits,withoutwaitingforanydata.
The reason for this is that the BEGIN keyword only applies the specified script before it
processesanydata.Ifyouwanttoprocessdatawithanormalprogramscript,youmust
definetheprogramusinganotherscriptsection:
$catdata3.txt
Line1
Line2
Line3
$
$gawk‘BEGIN{print”Thedata3FileContents:”}
>{print$0}’data3.txt
Thedata3FileContents:
Line1
Line2
Line3
$
NowaftergawkexecutestheBEGINscript,itusesthesecondscripttoprocessanyfiledata.
Becarefulwhendoingthis;bothofthescriptsarestillconsideredonetextstringonthe
gawkcommandline.Youneedtoplaceyoursinglequotationmarksaccordingly.
Runningscriptsafterprocessingdata
LiketheBEGINkeyword,theENDkeywordallowsyoutospecifyaprogramscriptthatgawk
executesafterreadingthedata:
$gawk‘BEGIN{print“Thedata3FileContents:”}
>{print$0}
>END{print“EndofFile”}’data3.txt
Thedata3FileContents:
Line1
Line2
Line3
EndofFile
$
Whenthegawkprogramisfinishedprintingthefilecontents,itexecutesthecommandsin
the END script. This is a great technique to use to add footer data to reports after all the
normaldatahasbeenprocessed.
Youcanputalltheseelementstogetherintoanicelittleprogramscriptfiletocreateafull
reportfromasimpledatafile:
$catscript4.gawk
BEGIN{
print“Thelatestlistofusersandshells”
print”UserID\tShell”
print“––—\t––-”
FS=”:”
}
{
print$1“\t“$7
}
END{
print“Thisconcludesthelisting”
}
$
Thisscriptusesthe BEGINscripttocreateaheadersectionforthereport.Italsodefinesa
specialvariablecalled FS.Thisisyetanotherwaytodefinethefieldseparationcharacter.
This way, you don’t have to depend on the script’s user to define the field separation
characterinthecommandlineoptions.
Here’sasomewhattruncatedoutputfromrunningthisgawkprogramscript:
$gawk-fscript4.gawk/etc/passwd
Thelatestlistofusersandshells
UserIDShell
––—––root/bin/bash
bin/sbin/nologin
daemon/sbin/nologin
[…]
Christine/bin/bash
mysql/bin/bash
Samantha/bin/bash
Timothy/bin/bash
Thisconcludesthelisting
$
As expected, the BEGIN script created the header text, the program script processed the
information from the specified data file (the /etc/passwd file), and the END script
produced the footer text. The ∖t within the print command produces some nicely
formattedtabbedoutput.
This gives you a small taste of the power available when you use simple gawk scripts.
Chapter 22 describes some more basic programming principles available for your gawk
scripts,alongwithsomeevenmoreadvancedprogrammingconceptsyoucanuseinyour
gawk program scripts to create professional looking reports from even the most cryptic
datafiles.
CommandingatthesedEditorBasics
The key to successfully using the sed editor is to know its myriad of commands and
formats,whichhelpyoutocustomizeyourtextediting.Thissectiondescribessomeofthe
basic commands and features you can incorporate into your script to start using the sed
editor.
Introducingmoresubstitutionoptions
You’vealreadyseenhowtousethescommandtosubstitutenewtextforthetextinaline.
However, a few additional options are available for the substitute command that can
helpmakeyourlifeeasier.
Substitutingflags
There’sacaveattohowthe substitutecommandreplacesmatchingpatternsinthetext
string.Watchwhathappensinthisexample:
$catdata4.txt
Thisisatestofthetestscript.
Thisisthesecondtestofthetestscript.
$
$sed‘s/test/trial/’data4.txt
Thisisatrialofthetestscript.
Thisisthesecondtrialofthetestscript.
$
Thesubstitutecommandworksfineinreplacingtextinmultiplelines,butbydefault,it
replacesonlythefirstoccurrenceineachline.Togetthe substitutecommandtowork
ondifferentoccurrencesofthetext,youmustuseasubstitutionflag.Thesubstitutionflag
issetafterthesubstitutioncommandstrings:
s/pattern/replacement/flags
Fourtypesofsubstitutionflagsareavailable:
Anumber,indicatingthepatternoccurrenceforwhichnewtextshouldbesubstituted
g,indicatingthatnewtextshouldbesubstitutedforalloccurrencesoftheexisting
text
p,indicatingthatthecontentsoftheoriginallineshouldbeprinted
wfile,whichmeanstowritetheresultsofthesubstitutiontoafile
Inthefirsttypeofsubstitution,youcanspecifywhichoccurrenceofthematchingpattern
thesededitorshouldsubstitutenewtextfor:
$sed‘s/test/trial/2’data4.txt
Thisisatestofthetrialscript.
Thisisthesecondtestofthetrialscript.
$
As a result of specifying a 2 as the substitution flag, the sed editor replaces the pattern
onlyinthesecondoccurrenceineachline.The gsubstitutionflagenablesyoutoreplace
everyoccurrenceofthepatterninthetext:
$sed‘s/test/trial/g’data4.txt
Thisisatrialofthetrialscript.
Thisisthesecondtrialofthetrialscript.
$
The p substitution flag prints a line that contains a matching pattern in the substitute
command.Thisismostoftenusedinconjunctionwiththe-nsedoption:
$catdata5.txt
Thisisatestline.
Thisisadifferentline.
$
$sed-n‘s/test/trial/p’data5.txt
Thisisatrialline.
$
The -n option suppresses output from the sed editor. However, the p substitution flag
outputs any line that has been modified. Using the two in combination produces output
onlyforlinesthathavebeenmodifiedbythesubstitutecommand.
Thewsubstitutionflagproducesthesameoutputbutstorestheoutputinthespecifiedfile:
$sed‘s/test/trial/wtest.txt’data5.txt
Thisisatrialline.
Thisisadifferentline.
$
$cattest.txt
Thisisatrialline.
$
Thenormaloutputofthe sededitorappearsin STDOUT,butonlythelinesthatincludethe
matchingpatternarestoredinthespecifiedoutputfile.
Replacingcharacters
Sometimes, you run across characters in text strings that aren’t easy to use in the
substitutionpattern.OnepopularexampleintheLinuxworldistheforwardslash(/).
Substitutingpathnamesinafilecangetawkward.Forexample,ifyouwantedtosubstitute
theCshellforthebashshellinthe/etc/passwdfile,you’dhavetodothis:
$sed‘s/\/bin\/bash/\/bin\/csh/’/etc/passwd
Because the forward slash is used as the string delimiter, you must use a backslash to
escapeitifitappearsinthepatterntext.Thisoftenleadstoconfusionandmistakes.
To solve this problem, the sed editor allows you to select a different character for the
stringdelimiterinthesubstitutecommand:
$sed‘s!/bin/bash!/bin/csh!’/etc/passwd
In this example, the exclamation point is used for the string delimiter, making the
pathnamesmucheasiertoreadandunderstand.
Usingaddresses
Bydefault,thecommandsyouuseinthe sededitorapplytoalllinesofthetextdata.If
youwanttoapplyacommandonlytoaspecificlineoragroupoflines,youmustuseline
addressing.
Therearetwoformsoflineaddressinginthesededitor:
Anumericrangeoflines
Atextpatternthatfiltersoutaline
Bothformsusethesameformatforspecifyingtheaddress:
[address]command
Youcanalsogroupmorethanonecommandtogetherforaspecificaddress:
address{
command1
command2
command3
}
The sed editor applies each of the commands you specify only to lines that match the
addressspecified.Thissectiondemonstratesusingbothoftheseaddressingtechniquesin
yoursededitorscripts.
Addressingthenumericline
When using numeric line addressing, you reference lines using their line position in the
textstream.The sededitorassignsthefirstlineinthetextstreamaslinenumberoneand
continuessequentiallyforeachnewline.
Theaddressyouspecifyinthecommandcanbeasinglelinenumberorarangeoflines
specified by a starting line number, a comma, and an ending line number. Here’s an
exampleofspecifyingalinenumbertowhichthesedcommandwillbeapplied:
$sed‘2s/dog/cat/’data1.txt
Thequickbrownfoxjumpsoverthelazydog
Thequickbrownfoxjumpsoverthelazycat
Thequickbrownfoxjumpsoverthelazydog
Thequickbrownfoxjumpsoverthelazydog
$
Thesededitormodifiedthetextonlyinlinetwopertheaddressspecified.Here’sanother
example,thistimeusingarangeoflineaddresses:
$sed‘2,3s/dog/cat/’data1.txt
Thequickbrownfoxjumpsoverthelazydog
Thequickbrownfoxjumpsoverthelazycat
Thequickbrownfoxjumpsoverthelazycat
Thequickbrownfoxjumpsoverthelazydog
$
Ifyouwanttoapplyacommandtoagroupoflinesstartingatsomepointwithinthetext,
butcontinuingtotheendofthetext,youcanusethespecialaddress,thedollarsign:
$sed‘2,$s/dog/cat/’data1.txt
Thequickbrownfoxjumpsoverthelazydog
Thequickbrownfoxjumpsoverthelazycat
Thequickbrownfoxjumpsoverthelazycat
Thequickbrownfoxjumpsoverthelazycat
$
Becauseyoumaynotknowhowmanylinesofdataareinthetext,thedollarsignoften
comesinhandy.
Usingtextpatternfilters
The other method of restricting which lines a command applies to is a bit more
complicated.The sededitorallowsyoutospecifyatextpatternthatitusestofilterlines
forthecommand.Thisistheformat:
/pattern/command
Youmustencapsulatethepatternyouspecifyinforwardslashes.The sededitorapplies
thecommandonlytolinesthatcontainthetextpatternyouspecify.
Forexample,ifyouwanttochangethedefaultshellforonlytheuserSamantha,you’duse
thesedcommand:
$grepSamantha/etc/passwd
Samantha:x:502:502::/home/Samantha:/bin/bash
$
$sed‘/Samantha/s/bash/csh/’/etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
[…]
Christine:x:501:501:ChristineB:/home/Christine:/bin/bash
Samantha:x:502:502::/home/Samantha:/bin/csh
Timothy:x:503:503::/home/Timothy:/bin/bash
$
Thecommandwasappliedonlytothelinewiththematchingtextpattern.Althoughusing
afixedtextpatternmaybeusefulforfilteringspecificvalues,asinthe useridexample,
it’s somewhat limited in what you can do with it. The sed editor uses a feature called
regular expressions in text patterns to allow you to create patterns that get pretty
involved.
Regular expressions allow you to create advanced text pattern–matching formulas to
match all sorts of data. These formulas combine a series of wildcard characters, special
characters,andfixedtextcharacterstoproduceaconcisepatternthatcanmatchjustabout
any text situation. Regular expressions are one of the scarier parts of shell script
programming,andChapter20coversthemingreatdetail.
Groupingcommands
If you need to perform more than one command on an individual line, group the
commands together using braces. The sed editor processes each command listed on the
addressline(s):
$sed‘2{
>s/fox/elephant/
>s/dog/cat/
>}’data1.txt
Thequickbrownfoxjumpsoverthelazydog.
Thequickbrownelephantjumpsoverthelazycat.
Thequickbrownfoxjumpsoverthelazydog.
Thequickbrownfoxjumpsoverthelazydog.
$
Both commands are processed against the address. And of course, you can specify an
addressrangebeforethegroupedcommands:
$sed‘3,${
>s/brown/green/
>s/lazy/active/
>}’data1.txt
Thequickbrownfoxjumpsoverthelazydog.
Thequickbrownfoxjumpsoverthelazydog.
Thequickgreenfoxjumpsovertheactivedog.
Thequickgreenfoxjumpsovertheactivedog.
$
Thesededitorappliesallthecommandstoallthelinesintheaddressrange.
Deletinglines
Thetextsubstitutioncommandisn’ttheonlycommandavailableinthe sededitor.Ifyou
needtodeletespecificlinesoftextinatextstream,youcanusethedeletecommand.
The delete command, d, pretty much does what it says. It deletes any text lines that
matchtheaddressingschemesupplied.Becarefulwiththe deletecommand,becauseif
youforgettoincludeanaddressingscheme,allthelinesaredeletedfromthestream:
$catdata1.txt
Thequickbrownfoxjumpsoverthelazydog
Thequickbrownfoxjumpsoverthelazydog
Thequickbrownfoxjumpsoverthelazydog
Thequickbrownfoxjumpsoverthelazydog
$
$sed‘d’data1.txt
$
Thedeletecommandisobviouslymostusefulwhenusedinconjunctionwithaspecified
address.Thisallowsyoutodeletespecificlinesoftextfromthedatastream,eitherbyline
number:
$catdata6.txt
Thisislinenumber1.
Thisislinenumber2.
Thisislinenumber3.
Thisislinenumber4.
$
$sed‘3d’data6.txt
Thisislinenumber1.
Thisislinenumber2.
Thisislinenumber4.
$
orbyaspecificrangeoflines:
$sed‘2,3d’data6.txt
Thisislinenumber1.
Thisislinenumber4.
$
orbyusingthespecialend-of-filecharacter:
$sed‘3,$d’data6.txt
Thisislinenumber1.
Thisislinenumber2.
$
Thepattern-matchingfeatureofthesededitoralsoappliestothedeletecommand:
$sed‘/number1/d’data6.txt
Thisislinenumber2.
Thisislinenumber3.
Thisislinenumber4.
$
Thesededitorremovesthelinecontainingtextthatmatchesthepatternyouspecify.
Note
Rememberthatthesededitordoesn’ttouchtheoriginalfile.Anylinesyoudeleteare
onlygonefromtheoutputofthesededitor.Theoriginalfilestillcontainsthe
“deleted”lines.
Youcanalsodeletearangeoflinesusingtwotextpatterns,butbecarefulifyoudothis.
The first pattern you specify “turns on” the line deletion, and the second pattern “turns
off” the line deletion. The sed editor deletes any lines between the two specified lines
(includingthespecifiedlines):
$sed‘/1/,/3/d’data6.txt
Thisislinenumber4.
$
Inaddition,youmustbecarefulbecausethedeletefeature“turnson”wheneverthe sed
editordetectsthestartpatterninthedatastream.Thismayproduceanunexpectedresult:
$catdata7.txt
Thisislinenumber1.
Thisislinenumber2.
Thisislinenumber3.
Thisislinenumber4.
Thisislinenumber1again.
Thisistextyouwanttokeep.
Thisisthelastlineinthefile.
$
$sed‘/1/,/3/d’data7.txt
Thisislinenumber4.
$
The second occurrence of a line with the number 1 in it triggered the deletecommand
again, deleting the rest of the lines in the data stream, because the stop pattern wasn’t
recognized.Ofcourse,theotherobviousproblemoccursifyouspecifyastoppatternthat
neverappearsinthetext:
$sed‘/1/,/5/d’data7.txt
$
Becausethedeletefeatures“turnedon”atthefirstpatternmatch,butneverfoundtheend
patternmatch,theentiredatastreamwasdeleted.
Insertingandappendingtext
Asyouwouldexpect,likeanyothereditor,thesededitorallowsyoutoinsertandappend
textlinestothedatastream.Thedifferencebetweenthetwoactionscanbeconfusing:
Theinsertcommand(i)addsanewlinebeforethespecifiedline.
Theappendcommand(a)addsanewlineafterthespecifiedline.
What is confusing about these two commands is their formats. You can’t use these
commands on a single command line. You must specify the line to insert or append the
linetoinsertonaseparatelinebyitself.Here’stheformatfordoingthis:
sed‘[address]command\
newline’
Thetextinnewlineappearsinthesededitoroutputintheplaceyouspecify.Remember
thatwhenyouusetheinsertcommand,thetextappearsbeforethedatastreamtext:
$echo“TestLine2”|sed‘i\TestLine1’
TestLine1
TestLine2
$
Andwhenyouusetheappendcommand,thetextappearsafterthedatastreamtext:
$echo“TestLine2”|sed‘a\TestLine1’
TestLine2
TestLine1
$
When you use the sed editor from the command line interface prompt, you get the
secondary prompt to enter the new line of data. You must complete the sed editor
command on this line. After you enter the ending single quotation mark, the bash shell
processesthecommand:
$echo“TestLine2”|sed‘i\
>TestLine1’
TestLine1
TestLine2
$
Thisworkswellforaddingtextbeforeorafterthetextinthedatastream,butwhatabout
addingtextinsidethedatastream?
Toinsertorappenddatainsidethedatastreamlines,youmustuseaddressingtotellthe
sededitorwhereyouwantthedatatoappear.Youcanspecifyonlyasinglelineaddress
when using these commands. You can match either a numeric line number or a text
pattern, but you cannot use a range of addresses. This is logical, because you can only
insertorappendbeforeorafterasingleline,andnotarangeoflines.
Here’sanexampleofinsertinganewlinebeforeline3inthedatastream:
$sed‘3i\
>Thisisaninsertedline.’data6.txt
Thisislinenumber1.
Thisislinenumber2.
Thisisaninsertedline.
Thisislinenumber3.
Thisislinenumber4.
$
Here’sanexampleofappendinganewlineafterline3inthedatastream:
$sed‘3a\
>Thisisanappendedline.’data6.txt
Thisislinenumber1.
Thisislinenumber2.
Thisislinenumber3.
Thisisanappendedline.
Thisislinenumber4.
$
Thisusesthesameprocessasthe insertcommand;itjustplacesthenewtextlineafter
thespecifiedlinenumber.Ifyouhaveamultilinedatastream,andyouwanttoappenda
newlineoftexttotheendofadatastream,justusethedollarsign,whichrepresentsthe
lastlineofdata:
$sed‘$a\
>Thisisanewlineoftext.’data6.txt
Thisislinenumber1.
Thisislinenumber2.
Thisislinenumber3.
Thisislinenumber4.
Thisisanewlineoftext.
$
Thesameideaappliesifyouwanttoaddanewlineatthebeginningofthedatastream.
Justinsertanewlinebeforelinenumberone.
Toinsertorappendmorethanonelineoftext,youmustuseabackslashoneachlineof
newtextuntilyoureachthelasttextlinewhereyouwanttoinsertorappendtext:
$sed‘1i\
>Thisisonelineofnewtext.\
>Thisisanotherlineofnewtext.’data6.txt
Thisisonelineofnewtext.
Thisisanotherlineofnewtext.
Thisislinenumber1.
Thisislinenumber2.
Thisislinenumber3.
Thisislinenumber4.
$
Bothofthespecifiedlinesareaddedtothedatastream.
Changinglines
The change command allows you to change the contents of an entire line of text in the
datastream.Itworksthesamewayastheinsertandappendcommands,inthatyoumust
specifythenewlineseparatelyfromtherestofthesedcommand:
$sed‘3c\
>Thisisachangedlineoftext.’data6.txt
Thisislinenumber1.
Thisislinenumber2.
Thisisachangedlineoftext.
Thisislinenumber4.
$
Inthisexample,the sededitorchangesthetextinlinenumber3.Youcanalsouseatext
patternfortheaddress:
$sed‘/number3/c\
>Thisisachangedlineoftext.’data6.txt
Thisislinenumber1.
Thisislinenumber2.
Thisisachangedlineoftext.
Thisislinenumber4.
$
The text pattern change command changes any line of text in the data stream that it
matches.
$catdata8.txt
Thisislinenumber1.
Thisislinenumber2.
Thisislinenumber3.
Thisislinenumber4.
Thisislinenumber1again.
Thisisyetanotherline.
Thisisthelastlineinthefile.
$
$sed‘/number1/c\
>Thisisachangedlineoftext.’data8.txt
Thisisachangedlineoftext.
Thisislinenumber2.
Thisislinenumber3.
Thisislinenumber4.
Thisisachangedlineoftext.
Thisisyetanotherline.
Thisisthelastlineinthefile.
$
Youcanuseanaddressrangeinthechangecommand,buttheresultsmaynotbewhatyou
expect:
$sed‘2,3c\
>Thisisanewlineoftext.’data6.txt
Thisislinenumber1.
Thisisanewlineoftext.
Thisislinenumber4.
$
Insteadofchangingbothlineswiththetext,the sededitorusesthesinglelineoftextto
replacebothlines.
Transformingcharacters
The transform command (y) is the only sed editor command that operates on a single
character.Thetransformcommandusestheformat:
[address]y/inchars/outchars/
The transform command performs a one-to-one mapping of the inchars and the
outchars values. The first character in inchars is converted to the first character in
outchars. The second character in inchars is converted to the second character in
outchars.Thismappingcontinuesthroughoutthelengthofthespecifiedcharacters.Ifthe
incharsandoutcharsarenotthesamelength,thesededitorproducesanerrormessage.
Here’sasimpleexampleofusingthetransformcommand:
$sed‘y/123/789/’data8.txt
Thisislinenumber7.
Thisislinenumber8.
Thisislinenumber9.
Thisislinenumber4.
Thisislinenumber7again.
Thisisyetanotherline.
Thisisthelastlineinthefile.
$
Asyoucanseefromtheoutput,eachinstanceofthecharactersspecifiedinthe inchars
patternhasbeenreplacedbythecharacterinthesamepositionintheoutcharspattern.
The transformcommandisaglobalcommand;thatis,itperformsthetransformationon
anycharacterfoundinthetextlineautomatically,withoutregardtotheoccurrence:
$echo“This1isatestof1try.”|sed‘y/123/456/’
This4isatestof4try.
$
Thesededitortransformedbothinstancesofthematchingcharacter1inthetextline.You
can’tlimitthetransformationtoaspecificoccurrenceofthecharacter.
Printingrevisited
The “Introducing more substitution options” section showed you how to use the p flag
with the substitution command to display lines that the sed editor changed. In addition,
threecommandsthatcanbeusedtoprintinformationfromthedatastream:
Thepcommandtoprintatextline
Theequalsign(=)commandtoprintlinenumbers
Thel(lowercaseL)commandtolistaline
Thefollowingsectionslookatthesethreeprintingcommandsinthesededitor.
Printinglines
Like the p flag in the substitution command, the p command prints a line in the sed
editoroutput.Onitsown,thiscommanddoesn’toffermuchexcitement:
$echo“thisisatest”|sed‘p’
thisisatest
thisisatest
$
Allitdoesisprintthedatatextthatyoualreadyknowisthere.Themostcommonusefor
theprintcommandisprintinglinesthatcontainmatchingtextfromatextpattern:
$catdata6.txt
Thisislinenumber1.
Thisislinenumber2.
Thisislinenumber3.
Thisislinenumber4.
$
$sed-n‘/number3/p’data6.txt
Thisislinenumber3.
$
Byusingthe-noptiononthecommandline,youcansuppressalltheotherlinesandprint
onlythelinethatcontainsthematchingtextpattern.
Youcanalsousethisasaquickwaytoprintasubsetoflinesinadatastream:
$sed-n‘2,3p’data6.txt
Thisislinenumber2.
Thisislinenumber3.
$
Youcanalsousethe printcommandwhenyouneedtoseealinebeforeitgetsaltered,
suchaswiththe substitutionor changecommand.Youcancreateascriptthatdisplays
thelinebeforeit’schanged:
$sed-n‘/3/{
>p
>s/line/test/p
>}’data6.txt
Thisislinenumber3.
Thisistestnumber3.
$
This sededitorcommandsearchesforlinesthatcontainthenumber3andexecutestwo
commands.First,thescriptusesthe pcommandtoprinttheoriginalversionoftheline;
thenitusesthe scommandtosubstitutetext,alongwiththe pflagtoprinttheresulting
text.Theoutputshowsboththeoriginallinetextandthenewlinetext.
Printinglinenumbers
The equal sign command prints the current line number for the line within the data
stream. Line numbers are determined by using the newline character in the data stream.
Eachtimeanewlinecharacterappearsinthedatastream,the sed editor assumes that it
terminatesalineoftext:
$catdata1.txt
Thequickbrownfoxjumpsoverthelazydog.
Thequickbrownfoxjumpsoverthelazydog.
Thequickbrownfoxjumpsoverthelazydog.
Thequickbrownfoxjumpsoverthelazydog.
$
$sed‘=’data1.txt
1
Thequickbrownfoxjumpsoverthelazydog.
2
Thequickbrownfoxjumpsoverthelazydog.
3
Thequickbrownfoxjumpsoverthelazydog.
4
Thequickbrownfoxjumpsoverthelazydog.
$
The sed editor prints the line number before the actual line of text. The equal sign
commandcomesinhandyifyou’researchingforaspecifictextpatterninthedatastream:
$sed-n‘/number4/{
>=
>p
>}’data6.txt
4
Thisislinenumber4.
$
Byusingthe -noption,youcanhavethe sededitordisplayboththelinenumberandtext
forthelinethatcontainsthematchingtextpattern.
Listinglines
The listcommand(l)allowsyoutoprintboththetextandnonprintablecharactersina
data stream. Any nonprintable characters are shown using either their octal values,
preceded by a backslash or the standard C-style nomenclature for common nonprintable
characters,suchas∖tfortabcharacters:
$catdata9.txt
Thislinecontainstabs.
$
$sed-n‘l’data9.txt
This\tline\tcontains\ttabs.$
$
Thetabcharacterlocationsareshownwiththe∖tnomenclature.Thedollarsignattheend
of the line indicates the newline character. If you have a data stream that contains an
escapecharacter,thelistcommanddisplaysitusingtheoctalcodeifnecessary:
$catdata10.txt
Thislinecontainsanescapecharacter.
$
$sed-n‘l’data10.txt
Thislinecontainsanescapecharacter.\a$
$
Thedata10.txtfilecontainsanescapecontrolcode,whichgeneratesabellsound.When
youusethe catcommandtodisplaythetextfile,youdon’tseetheescapecontrolcode;
you just hear the sound (if your speakers are turned on). However, using the list
command,youcandisplaytheescapecontrolcodeused.
Usingfileswithsed
The substitutioncommandcontainsflagsthatallowyoutoworkwithfiles.Thereare
alsoregularsededitorcommandsthatletyoudothatwithouthavingtosubstitutetext.
Writingtoafile
Thewcommandisusedtowritelinestoafile.Here’stheformatforthewcommand:
[address]wfilename
Thefilenamecanbespecifiedaseitherarelativeorabsolutepathname,butineithercase,
thepersonrunningthesededitormusthavewritepermissionsforthefile.Theaddresscan
beanytypeofaddressingmethodusedinsed,suchasasinglelinenumber,atextpattern,
orarangeoflinenumbersortextpatterns.
Here’sanexamplethatprintsonlythefirsttwolinesofadatastreamtoatextfile:
$sed‘1,2wtest.txt’data6.txt
Thisislinenumber1.
Thisislinenumber2.
Thisislinenumber3.
Thisislinenumber4.
$
$cattest.txt
Thisislinenumber1.
Thisislinenumber2.
$
Ofcourse,ifyoudon’twantthelinestodisplayon STDOUT,youcanusethe -noptionfor
thesedcommand.
Thisisagreattooltouseifyouneedtocreateadatafilefromamasterfileonthebasisof
commontextvalues,suchasthoseinamailinglist:
$catdata11.txt
Blum,RBrowncoat
McGuiness,AAlliance
Bresnahan,CBrowncoat
Harken,CAlliance
$
$sed-n‘/Browncoat/wBrowncoats.txt’data11.txt
$
$catBrowncoats.txt
Blum,RBrowncoat
Bresnahan,CBrowncoat
$
Thesededitorwritestoadestinationfileonlythedatalinesthatcontainthetextpattern.
Readingdatafromafile
You’vealreadyseenhowtoinsertdataintoandappendtexttoadatastreamfromthe sed
commandline.The readcommand(r)allowsyoutoinsertdatacontainedinaseparate
file.
Here’stheformatofthereadcommand:
[address]rfilename
The filenameparameterspecifieseitheranabsoluteorrelativepathnameforthefilethat
containsthedata.Youcan’tusearangeofaddressesforthereadcommand.Youcanonly
specifyasinglelinenumberortextpatternaddress.The sededitorinsertsthetextfrom
thefileaftertheaddress.
$catdata12.txt
Thisisanaddedline.
Thisisthesecondaddedline.
$
$sed‘3rdata12.txt’data6.txt
Thisislinenumber1.
Thisislinenumber2.
Thisislinenumber3.
Thisisanaddedline.
Thisisthesecondaddedline.
Thisislinenumber4.
$
The sed editor inserts into the data stream all the text lines in the data file. The same
techniqueworkswhenusingatextpatternaddress:
$sed‘/number2/rdata12.txt’data6.txt
Thisislinenumber1.
Thisislinenumber2.
Thisisanaddedline.
Thisisthesecondaddedline.
Thisislinenumber3.
Thisislinenumber4.
$
Ifyouwanttoaddtexttotheendofadatastream,justusethedollarsignaddresssymbol:
$sed‘$rdata12.txt’data6.txt
Thisislinenumber1.
Thisislinenumber2.
Thisislinenumber3.
Thisislinenumber4.
Thisisanaddedline.
Thisisthesecondaddedline.
$
A cool application of the read command is to use it in conjunction with a delete
command to replace a placeholder in a file with data from another file. For example,
supposethatyouhadaformstoredinatextfilethatlookedlikethis:
$catnotice.std
Wouldthefollowingpeople:
LIST
pleasereporttotheship’scaptain.
$
TheformletterusesthegenericplaceholderLISTinplaceofalistofpeople.Toinsertthe
list of people after the placeholder, you just use the read command. However, this still
leavestheplaceholdertextintheoutput.Toremovethat,justusethe deletecommand.
Theresultlookslikethis:
$sed‘/LIST/{
>rdata11.txt
>d
>}’notice.std
Wouldthefollowingpeople:
Blum,RBrowncoat
McGuiness,AAlliance
Bresnahan,CBrowncoat
Harken,CAlliance
pleasereporttotheship’scaptain.
$
Nowtheplaceholdertextisreplacedwiththelistofnamesfromthedatafile.
Summary
Shellscriptscandolotsofworkontheirown,butit’softendifficulttomanipulatedata
withjustashellscript.Linuxprovidestwohandyutilitiestohelpwithhandlingtextdata.
The sededitorisastreameditorthatquicklyprocessesdataontheflyasitreadsit.You
mustprovidethesededitorwithalistofeditingcommands,whichitappliestothedata.
ThegawkprogramisautilityfromtheGNUorganizationthatmimicsandexpandsonthe
functionalityoftheUnixawkprogram.Thegawkprogramcontainsabuilt-inprogramming
languagethatyoucanusetowritescriptstohandleandprocessdata.Youcanusethegawk
programtoextractdataelementsfromlargedatafilesandoutputtheminjustaboutany
formatyoudesire.Thismakesprocessinglargelogfilesasnap,aswellascreatingcustom
reportsfromdatafiles.
Acrucialelementofusingboththesedandgawkprogramsisknowinghowtouseregular
expressions.Regularexpressionsarekeytocreatingcustomizedfiltersforextractingand
manipulatingdataintextfiles.Thenextchapterdivesintotheoftenmisunderstoodworld
ofregularexpressions,showingyouhowtobuildregularexpressionsformanipulatingall
typesofdata.
Chapter20
RegularExpressions
InThisChapter
1. Definingregularexpressions
2. Lookingatthebasics
3. Extendingourpatterns
4. Creatingexpressions
Thekeytosuccessfullyworkingwiththesededitorandthegawkprograminyour
shellscriptisyourcomfortusingregularexpressions.Thisisnotalwaysaneasy
thingtodo,becausetryingtofilterspecificdatafromalargebatchofdatacan(and
oftendoes)getcomplicated.Thischapterdescribeshowtocreateregularexpressions
inboththesededitorandthegawkprogramthatcanfilteroutjustthedatayouneed.
WhatAreRegularExpressions?
Thefirststeptounderstandingregularexpressionsistodefinejustexactlywhattheyare.
ThissectionexplainswhataregularexpressionisanddescribeshowLinuxusesregular
expressions.
Adefinition
AregularexpressionisapatterntemplateyoudefinethataLinuxutilityusestofiltertext.
A Linux utility (such as the sed editor or the gawk program) matches the regular
expressionpatternagainstdataasthatdataflowsintotheutility.Ifthedatamatchesthe
pattern, it’s accepted for processing. If the data doesn’t match the pattern, it’s rejected.
ThisisillustratedinFigure20.1.
Figure20.1Matchingdataagainstaregularexpressionpattern
Theregularexpressionpatternmakesuseofwildcardcharacterstorepresentoneormore
charactersinthedatastream.ThereareplentyofinstancesinLinuxwhereyoucanspecify
a wildcard character to represent data you don’t know about. You’ve already seen an
example of using wildcard characters with the Linux ls command for listing files and
directories(seeChapter3).
Theasteriskwildcardcharacterallowsyoutolistonlyfilesthatmatchacertaincriteria.
Forexample:
$ls-alda*
-rw-r—r—1richrich45Nov2612:42data
-rw-r—r—1richrich25Dec412:40data.tst
-rw-r—r—1richrich180Nov2612:42data1
-rw-r—r—1richrich45Nov2612:44data2
-rw-r—r—1richrich73Nov2712:31data3
-rw-r—r—1richrich79Nov2814:01data4
-rw-r—r—1richrich187Dec409:45datatest
$
The da*parameterinstructsthe lscommandtolistonlythefileswhosenamestartswith
da.Therecanbeanynumberofcharactersafterthe dainthefilename(includingnone).
Thelscommandreadstheinformationregardingallthefilesinthedirectorybutdisplays
onlytheonesthatmatchthewildcardcharacter.
Regular expression wildcard patterns work in a similar way. The regular expression
patterncontainstextand/orspecialcharactersthatdefineatemplateforthesededitorand
the gawkprogramtofollowwhenmatchingdata.Youcanusedifferentspecialcharacters
inaregularexpressiontodefineaspecificpatternforfilteringdata.
Typesofregularexpressions
Thebiggestproblemwithusingregularexpressionsisthatthereisn’tjustonesetofthem.
Several different applications use different types of regular expressions in the Linux
environment. These include such diverse applications as programming languages (Java,
Perl,andPython),Linuxutilities(suchasthe sededitor,the gawkprogram,andthegrep
utility), and mainstream applications (such as the MySQL and PostgreSQL database
servers).
A regular expression is implemented using a regular expression engine. A regular
expressionengineistheunderlyingsoftwarethatinterpretsregularexpressionpatternsand
usesthosepatternstomatchtext.
TheLinuxworldhastwopopularregularexpressionengines:
ThePOSIXBasicRegularExpression(BRE)engine
ThePOSIXExtendedRegularExpression(ERE)engine
Most Linux utilities at a minimum conform to the POSIX BRE engine specifications,
recognizing all the pattern symbols it defines. Unfortunately, some utilities (such as the
sededitor)conformonlytoasubsetoftheBREenginespecifications.Thisisduetospeed
constraints,becausethesededitorattemptstoprocesstextinthedatastreamasquicklyas
possible.
The POSIX ERE engine is often found in programming languages that rely on regular
expressions for text filtering. It provides advanced pattern symbols as well as special
symbols for common patterns, such as matching digits, words, and alphanumeric
characters. The gawk program uses the ERE engine to process its regular expression
patterns.
Because there are so many different ways to implement regular expressions, it’s hard to
presentasingle,concisedescriptionofallthepossibleregularexpressions.Thefollowing
sections discuss the most commonly found regular expressions and demonstrate how to
usetheminthesededitorandgawkprogram.
DefiningBREPatterns
The most basic BRE pattern is matching text characters in a data stream. This section
demonstrateshowyoucandefinetextintheregularexpressionpatternandwhattoexpect
fromtheresults.
Plaintext
Chapter18demonstratedhowtousestandardtextstringsinthe sededitorandthe gawk
programtofilterdata.Here’sanexampletorefreshyourmemory:
$echo“Thisisatest”|sed-n‘/test/p’
Thisisatest
$echo“Thisisatest”|sed-n‘/trial/p’
$
$echo“Thisisatest”|gawk‘/test/{print$0}’
Thisisatest
$echo“Thisisatest”|gawk‘/trial/{print$0}’
$
Thefirstpatterndefinesasingleword,test.Thesededitorand gawkprogramscriptseach
use their own version of the print command to print any lines that match the regular
expressionpattern.Becausethe echostatementcontainstheword“test”inthetextstring,
the data stream text matches the defined regular expression pattern, and the sed editor
displaystheline.
Thesecondpatternagaindefinesjustasingleword,thistimetheword“trial.”Becausethe
echostatementtextstringdoesn’tcontainthatword,theregularexpressionpatterndoesn’t
match,soneitherthesededitornorthegawkprogramprintstheline.
You probably already noticed that the regular expression doesn’t care where in the data
streamthepatternoccurs.Italsodoesn’tmatterhowmanytimesthepatternoccurs.After
the regular expression can match the pattern anywhere in the text string, it passes the
stringalongtotheLinuxutilitythat’susingit.
Thekeyismatchingtheregularexpressionpatterntothedatastreamtext.It’simportantto
rememberthatregularexpressionsareextremelypickyaboutmatchingpatterns.Thefirst
ruletorememberisthatregularexpressionpatternsarecasesensitive.Thismeansthey’ll
matchonlythosepatternswiththepropercaseofcharacters:
$echo“Thisisatest”|sed-n‘/this/p’
$
$echo“Thisisatest”|sed-n‘/This/p’
Thisisatest
$
Thefirstattemptfailedtomatchbecausetheword“this”doesn’tappearinalllowercasein
the text string, while the second attempt, which uses the uppercase letter in the pattern,
workedjustfine.
Youdon’thavetolimityourselftowholewordsintheregularexpression.Ifthedefined
textappearsanywhereinthedatastream,theregularexpressionmatchesthefollowing:
$echo“Thebooksareexpensive”|sed-n‘/book/p’
Thebooksareexpensive
$
Even though the text in the data stream is books, the data in the stream contains the
regularexpressionbook,sotheregularexpressionpatternmatchesthedata.Ofcourse,if
youtrytheopposite,theregularexpressionfails:
$echo“Thebookisexpensive”|sed-n‘/books/p’
$
Thecompleteregularexpressiontextdidn’tappearinthedatastream,sothematchfailed
andthesededitordidn’tdisplaythetext.
Youalsodon’thavetolimityourselftosingletextwordsintheregularexpression.You
canincludespacesandnumbersinyourtextstringaswell:
$echo“Thisislinenumber1”|sed-n‘/ber1/p’
Thisislinenumber1
$
Spacesaretreatedjustlikeanyothercharacterintheregularexpression:
$echo“Thisislinenumber1”|sed-n‘/ber1/p’
$
Ifyoudefineaspaceintheregularexpression,itmustappearinthedatastream.Youcan
evencreatearegularexpressionpatternthatmatchesmultiplecontiguousspaces:
$catdata1
Thisisanormallineoftext.
Thisisalinewithtoomanyspaces.
$sed-n‘//p’data1
Thisisalinewithtoomanyspaces.
$
Thelinewithtwospacesbetweenwordsmatchestheregularexpressionpattern.Thisisa
greatwaytocatchspacingproblemsintextfiles!
Specialcharacters
Asyouusetextstringsinyourregularexpressionpatterns,there’ssomethingyouneedto
be aware of. There are a few exceptions when defining text characters in a regular
expression. Regular expression patterns assign a special meaning to a few characters. If
you try to use these characters in your text pattern, you won’t get the results you were
expecting.
Thesespecialcharactersarerecognizedbyregularexpressions:
.*[]^${}\+?|()
Asthechapterprogresses,you’llfindoutjustwhatthesespecialcharactersdoinaregular
expression. For now, however, just remember that you can’t use these characters by
themselvesinyourtextpattern.
Ifyouwanttouseoneofthespecialcharactersasatextcharacter,youneedtoescapeit.
When you escape the special characters, you add a special character in front of it to
indicate to the regular expression engine that it should interpret the next character as a
normaltextcharacter.Thespecialcharacterthatdoesthisisthebackslashcharacter(∖).
For example, if you want to search for a dollar sign in your text, just precede it with a
backslashcharacter:
$catdata2
Thecostis$4.00
$sed-n‘/\$/p’data2
Thecostis$4.00
$
Becausethebackslashisaspecialcharacter,ifyouneedtouseitinaregularexpression
pattern,youneedtoescapeitaswell,producingadoublebackslash:
$echo“\isaspecialcharacter”|sed-n‘/[[BackslashBackslash]]/p’
\isaspecialcharacter
$
Finally,althoughtheforwardslashisn’taregularexpressionspecialcharacter,ifyouuse
it in your regular expression pattern in the sed editor or the gawk program, you get an
error:
$echo“3/2”|sed-n‘///p’
sed:-eexpression#1,char2:Nopreviousregularexpression
$
Touseaforwardslash,youneedtoescapethataswell:
$echo“3/2”|sed-n‘/\//p’
3/2
$
Nowthesededitorcanproperlyinterprettheregularexpressionpattern,andalliswell.
Anchorcharacters
Asshowninthe“PlainText”section,bydefault,whenyouspecifyaregularexpression
pattern,ifthepatternappearsanywhereinthedatastream,itmatches.Youcanusetwo
specialcharacterstoanchorapatterntoeitherthebeginningortheendoflinesinthedata
stream.
Startingatthebeginning
Thecaretcharacter(∧)definesapatternthatstartsatthebeginningofalineoftextinthe
datastream.Ifthepatternislocatedanyplaceotherthanthestartofthelineoftext,the
regularexpressionpatternfails.
To use the caret character, you must place it before the pattern specified in the regular
expression:
$echo“Thebookstore”|sed-n‘/^book/p’
$
$echo“Booksaregreat”|sed-n‘/^Book/p’
Booksaregreat
$
Thecaretanchorcharacterchecksforthepatternatthebeginningofeachnewlineofdata,
asdeterminedbythenewlinecharacter:
$catdata3
Thisisatestline.
thisisanothertestline.
Alinethatteststhisfeature.
Yetmoretestingofthis
$sed-n‘/^this/p’data3
thisisanothertestline.
$
Aslongasthepatternappearsatthestartofanewline,thecaretanchorcatchesit.
Ifyoupositionthecaretcharacterinanyplaceotherthanatthebeginningofthepattern,it
actslikeanormalcharacterandnotasaspecialcharacter:
$echo“This^isatest”|sed-n‘/s^/p’
This^isatest
$
Becausethecaretcharacterislistedlastintheregularexpressionpattern,the sededitor
usesitasanormalcharactertomatchtext.
Note
Ifyouneedtospecifyaregularexpressionpatternusingonlythecaretcharacter,you
don’tneedtoescapeitwithabackslash.However,ifyouspecifythecaretcharacter
first,followedbyadditionaltextinthepattern,youneedtousetheescapecharacter
beforethecaretcharacter.
Lookingfortheending
Theoppositeoflookingforapatternatthestartofalineislookingforitattheendofa
line. The dollar sign ($) special character defines the end anchor. Add this special
characterafteratextpatterntoindicatethatthelineofdatamustendwiththetextpattern:
$echo“Thisisagoodbook”|sed-n‘/book$/p’
Thisisagoodbook
$echo“Thisbookisgood”|sed-n‘/book$/p’
$
Theproblemwithanendingtextpatternisthatyoumustbecarefulwhatyou’relooking
for:
$echo“Therearealotofgoodbooks”|sed-n‘/book$/p’
$
Makingtheword“book”pluralattheendofthelinemeansthatitnolongermatchesthe
regularexpressionpattern,eventhoughbookisinthedatastream.Thetextpatternmust
bethelastthingonthelineforthepatterntomatch.
Combininganchors
Insomecommonsituations,youcancombineboththestartandendanchoronthesame
line.Inthefirstsituation,supposeyouwanttolookforalineofdatacontainingonlya
specifictextpattern:
$catdata4
thisisatestofusingbothanchors
Isaidthisisatest
thisisatest
I’msurethisisatest.
$sed-n‘/^thisisatest$/p’data4
thisisatest
$
Thesededitorignoresthelinesthatincludeothertextbesidesthespecifiedtext.
Thesecondsituationmayseemalittleoddatfirstbutisextremelyuseful.Bycombining
both anchors in a pattern with no text, you can filter blank lines from the data stream.
Considerthisexample:
$catdata5
Thisisonetestline.
Thisisanothertestline.
$sed‘/^$/d’data5
Thisisonetestline.
Thisisanothertestline.
$
Theregularexpressionpatternthatisdefinedlooksforlinesthathavenothingbetweenthe
start and end of the line. Because blank lines contain no text between the two newline
characters, they match the regular expression pattern. The sed editor uses the d delete
commandtodeletelinesthatmatchtheregularexpressionpattern,thusremovingallblank
linesfromthetext.Thisisaneffectivewaytoremoveblanklinesfromdocuments.
Thedotcharacter
Thedotspecialcharacterisusedtomatchanysinglecharacterexceptanewlinecharacter.
Thedotcharactermustmatchacharacter,however;ifthere’snocharacterintheplaceof
thedot,thenthepatternfails.
Let’slookatafewexamplesofusingthedotcharacterinaregularexpressionpattern:
$catdata6
Thisisatestofaline.
Thecatissleeping.
Thatisaverynicehat.
Thistestisatlinefour.
atteno’clockwe’llgohome.
$sed-n‘/.at/p’data6
Thecatissleeping.
Thatisaverynicehat.
Thistestisatlinefour.
$
You should be able to figure out why the first line failed and why the second and third
linespassed.Thefourthlineisalittletricky.Noticethatwematchedtheat,butthere’sno
characterinfrontofittomatchthedotcharacter.Ah,butthereis!Inregularexpressions,
spacescountascharacters,sothespaceinfrontofthe atmatchesthepattern.Thefifth
lineprovesthis,byputtingtheatinthefrontoftheline,whichfailstomatchthepattern.
Characterclasses
Thedotspecialcharacterisgreatformatchingacharacterpositionagainstanycharacter,
butwhatifyouwanttolimitwhatcharacterstomatch?Thisiscalledacharacterclassin
regularexpressions.
Youcandefineaclassofcharactersthatwouldmatchapositioninatextpattern.Ifoneof
thecharactersfromthecharacterclassisinthedatastream,itmatchesthepattern.
To define a character class, you use square brackets. The brackets should contain any
characteryouwanttoincludeintheclass.Youthenusetheentireclasswithinapattern
justlikeanyotherwildcardcharacter.Thistakesalittlegettingusedtoatfirst,butafter
youcatchon,itcangeneratesomeprettyamazingresults.
Thefollowingisanexampleofcreatingacharacterclass:
$sed-n‘/[ch]at/p’data6
Thecatissleeping.
Thatisaverynicehat.
$
Using the same data file as in the dot special character example, we came up with a
differentresult.Thistimewemanagedtofilteroutthelinethatjustcontainedthewordat.
The only words that match this pattern are cat and hat. Also notice that the line that
startedwith atdidn’tmatchaswell.Theremustbeacharacterinthecharacterclassthat
matchestheappropriateposition.
Characterclassescomeinhandyifyou’renotsurewhichcaseacharacterisin:
$echo“Yes”|sed-n‘/[Yy]es/p’
Yes
$echo“yes”|sed-n‘/[Yy]es/p’
yes
$
Youcanusemorethanonecharacterclassinasingleexpression:
$echo“Yes”|sed-n‘/[Yy][Ee][Ss]/p’
Yes
$echo“yEs”|sed-n‘/[Yy][Ee][Ss]/p’
yEs
$echo“yeS”|sed-n‘/[Yy][Ee][Ss]/p’
yeS
$
Theregularexpressionusedthreecharacterclassestocoverbothloweranduppercases
forallthreecharacterpositions.
Characterclassesdon’thavetocontainjustletters;youcanusenumbersinthemaswell:
$catdata7
Thislinedoesn’tcontainanumber.
Thislinehas1numberonit.
Thislineanumber2onit.
Thislinehasanumber4onit.
$sed-n‘/[0123]/p’data7
Thislinehas1numberonit.
Thislineanumber2onit.
$
The regular expression pattern matches any lines that contain the numbers 0, 1, 2, or 3.
Anyothernumbersareignored,asarelineswithoutnumbersinthem.
You can combine character classes to check for properly formatted numbers, such as
phonenumbersandZIPcodes.However,whenyou’retryingtomatchaspecificformat,
youmustbecareful.Here’sanexampleofaZIPcodematchgonewrong:
$catdata8
60633
46201
223001
4353
22203
$sed-n‘
>/[0123456789][0123456789][0123456789][0123456789][0123456789]/p
>’data8
60633
46201
223001
22203
$
Thismightnothaveproducedtheresultyouwerethinkingof.Itdidafinejoboffiltering
outthenumberthatwastooshorttobeaZIPcode,becausethelastcharacterclassdidn’t
have a character to match against. However, it still passed the six-digit number, even
thoughweonlydefinedfivecharacterclasses.
Remember that the regular expression pattern can be found anywhere in the text of the
data stream. You may always have additional characters besides the matching pattern
characters.Ifyouwanttoensurethatyoumatchagainstonlyfivenumbers,youneedto
delineatethemsomehow,eitherwithspaces,orasinthisexample,byshowingthatthey’re
atthestartandendoftheline:
$sed-n‘
>/^[0123456789][0123456789][0123456789][0123456789][0123456789]$/p
>‘data8
60633
46201
22203
$
Nowthat’smuchbetter!Laterinthischapter,welookathowtosimplifythisevenfurther.
Oneextremelypopularuseforcharacterclassesisparsingwordsthatmightbemisspelled,
suchasdataenteredfromauserform.Youcaneasilycreateregularexpressionsthatcan
acceptcommonmisspellingsindata:
$catdata9
Ineedtohavesomemaintenencedoneonmycar.
I’llpaythatinaseperateinvoice.
AfterIpayforthemaintenancemycarwillbeasgoodasnew.
$sed-n‘
/maint[ea]n[ae]nce/p
/sep[ea]r[ea]te/p
‘data9
Ineedtohavesomemaintenencedoneonmycar.
I’llpaythatinaseperateinvoice.
AfterIpayforthemaintenancemycarwillbeasgoodasnew.
$
Thetwo sedprintcommandsinthisexampleutilizeregularexpressioncharacterclasses
to help catch the misspelled words, maintenance and separate, in the text. The same
regularexpressionpatternalsomatchestheproperlyspelledoccurrenceof“maintenance.”
Negatingcharacterclasses
Inregularexpressionpatterns,youcanalsoreversetheeffectofacharacterclass.Instead
oflookingforacharactercontainedintheclass,youcanlookforanycharacterthat’snot
intheclass.Todothat,justplaceacaretcharacteratthebeginningofthecharacterclass
range:
$sed-n‘/[^ch]at/p’data6
Thistestisatlinefour.
$
Bynegatingthecharacterclass,theregularexpressionpatternmatchesanycharacterthat’s
neither a c nor an h, along with the text pattern. Because the space character fits this
category,itpassedthepatternmatch.However,evenwiththenegation,thecharacterclass
must still match a character, so the line with the at in the start of the line still doesn’t
matchthepattern.
Usingranges
YoumayhavenoticedwhenIshowedtheZIPcodeexampleearlierthatitwassomewhat
awkwardhavingtolistallthepossibledigitsineachcharacterclass.Fortunately,youcan
useashortcutsoyoudon’thavetodothat.
Youcanusearangeofcharacterswithinacharacterclassbyusingthedashsymbol.Just
specifythefirstcharacterintherange,adash,andthenthelastcharacterintherange.The
regular expression includes any character that’s within the specified character range,
accordingtothecharactersetusedbytheLinuxsystem(seeChapter2).
NowyoucansimplifytheZIPcodeexamplebyspecifyingarangeofdigits:
$sed-n‘/^[0-9][0-9][0-9][0-9][0-9]$/p’data8
60633
46201
45902
$
Thatsavedlotsoftyping!Eachcharacterclassmatchesanydigitfrom0to9.Thepattern
failsifaletterispresentanywhereinthedata:
$echo“a8392”|sed-n‘/^[0-9][0-9][0-9][0-9][0-9]$/p’
$
$echo“1839a”|sed-n‘/^[0-9][0-9][0-9][0-9][0-9]$/p’
$
$echo“18a92”|sed-n‘/^[0-9][0-9][0-9][0-9][0-9]$/p’
$
Thesametechniqueworkswithletters:
$sed-n‘/[c-h]at/p’data6
Thecatissleeping.
Thatisaverynicehat.
$
Thenewpattern [c-h]atmatcheswordswherethefirstletterisbetweenthelettercand
theletterh.Inthiscase,thelinewithonlythewordatfailedtomatchthepattern.
Youcanalsospecifymultiple,non-continuousrangesinasinglecharacterclass:
$sed-n‘/[a-ch-m]at/p’data6
Thecatissleeping.
Thatisaverynicehat.
$
Thecharacterclassallowstherangesathroughc,andhthroughmtoappearbeforetheat
text.Thisrangewouldrejectanylettersbetweendandg:
$echo“I’mgettingtoofat.”|sed-n‘/[a-ch-m]at/p’
$
Thispatternrejectedthefattext,asitwasn’tinthespecifiedrange.
Specialcharacterclasses
In addition to defining your own character classes, the BRE contains special character
classesyoucanusetomatchagainstspecifictypesofcharacters.Table20.1describesthe
BREspecialcharactersyoucanuse.
Table20.1BRESpecialCharacterClasses
Class
[[:alpha:]]
[[:alnum:]]
[[:blank:]]
[[:digit:]]
[[:lower:]]
[[:print:]]
[[:punct:]]
[[:space:]]
[[:upper:]]
Description
Matchesanyalphabeticalcharacter,eitherupperorlowercase
Matchesanyalphanumericcharacter0–9,A–Z,ora–z
MatchesaspaceorTabcharacter
Matchesanumericaldigitfrom0through9
Matchesanylowercasealphabeticalcharactera–z
Matchesanyprintablecharacter
Matchesapunctuationcharacter
Matchesanywhitespacecharacter:space,Tab,NL,FF,VT,CR
MatchesanyuppercasealphabeticalcharacterA–Z
Youusethespecialcharacterclassesjustasyouwouldanormalcharacterclassinyour
regularexpressionpatterns:
$echo“abc”|sed-n‘/[[:digit:]]/p’
$
$echo“abc”|sed-n‘/[[:alpha:]]/p’
abc
$echo“abc123”|sed-n‘/[[:digit:]]/p’
abc123
$echo“Thisis,atest”|sed-n‘/[[:punct:]]/p’
Thisis,atest
$echo“Thisisatest”|sed-n‘/[[:punct:]]/p’
$
Usingthespecialcharacterclassesisaneasywaytodefineranges.Insteadofhavingto
usearange[0–9],youcanjustuse[[:digit:]].
Theasterisk
Placinganasteriskafteracharactersignifiesthatthecharactermustappearzeroormore
timesinthetexttomatchthepattern:
$echo“ik”|sed-n‘/ie*k/p’
ik
$echo“iek”|sed-n‘/ie*k/p’
iek
$echo“ieek”|sed-n‘/ie*k/p’
ieek
$echo“ieeek”|sed-n‘/ie*k/p’
ieeek
$echo“ieeeek”|sed-n‘/ie*k/p’
ieeeek
$
This pattern symbol is commonly used for handling words that have a common
misspellingorvariationsinlanguagespellings.Forexample,ifyouneedtowriteascript
thatmaybeusedineitherAmericanorBritishEnglish,youcouldwrite:
$echo“I’mgettingacolorTV”|sed-n‘/colou*r/p’
I’mgettingacolorTV
$echo“I’mgettingacolourTV”|sed-n‘/colou*r/p’
I’mgettingacolourTV
$
Theu*inthepatternindicatesthattheletterumayormaynotappearinthetexttomatch
the pattern. Similarly, if you know of a word that is commonly misspelled, you can
accommodateitbyusingtheasterisk:
$echo“Iateapotatoewithmylunch.”|sed-n‘/potatoe*/p’
Iateapotatoewithmylunch.
$echo“Iateapotatowithmylunch.”|sed-n‘/potatoe*/p’
Iateapotatowithmylunch.
$
Placing an asterisk next to the possible extra letter allows you to accept the misspelled
word.
Another handy feature is combining the dot special character with the asterisk special
character.Thiscombinationprovidesapatterntomatchanynumberofanycharacters.It’s
oftenusedbetweentwotextstringsthatmayormaynotappearnexttoeachotherinthe
datastream:
$echo“thisisaregularpatternexpression”|sed-n‘
>/regular.*expression/p’
thisisaregularpatternexpression
$
Usingthispattern,youcaneasilysearchformultiplewordsthatmayappearanywhereina
lineoftextinthedatastream.
Theasteriskcanalsobeappliedtoacharacterclass.Thisallowsyoutospecifyagroupor
rangeofcharactersthatcanappearmorethanonceinthetext:
$echo“bt”|sed-n‘/b[ae]*t/p’
bt
$echo“bat”|sed-n‘/b[ae]*t/p’
bat
$echo“bet”|sed-n‘/b[ae]*t/p’
bet
$echo“btt”|sed-n‘/b[ae]*t/p’
btt
$
$echo“baat”|sed-n‘/b[ae]*t/p’
baat
$echo“baaeeet”|sed-n‘/b[ae]*t/p’
baaeeet
$echo“baeeaeeat”|sed-n‘/b[ae]*t/p’
baeeaeeat
$echo“baakeeet”|sed-n‘/b[ae]*t/p’
$
Aslongastheaandecharactersappearinanycombinationbetweenthebandtcharacters
(includingnotappearingatall),thepatternmatches.Ifanyothercharacteroutsideofthe
definedcharacterclassappears,thepatternmatchfails.
ExtendedRegularExpressions
ThePOSIXEREpatternsincludeafewadditionalsymbolsthatareusedbysomeLinux
applications and utilities. The gawk program recognizes the ERE patterns, but the sed
editordoesn’t.
Caution
Rememberthattheregularexpressionenginesinthesededitorandthegawkprogram
aredifferent.Thegawkprogramcanusemostoftheextendedregularexpression
patternsymbols,anditcanprovidesomeadditionalfilteringcapabilitiesthatthesed
editordoesn’thave.However,becauseofthis,itisoftenslowerinprocessingdata
streams.
ThissectiondescribesthemorecommonlyfoundEREpatternsymbolsthatyoucanusein
yourgawkprogramscripts.
Thequestionmark
The question mark is similar to the asterisk, but with a slight twist. The question mark
indicatesthattheprecedingcharactercanappearzerooronetime,butthat’sall.Itdoesn’t
matchrepeatingoccurrencesofthecharacter:
$echo“bt”|gawk‘/be?t/{print$0}’
bt
$echo“bet”|gawk‘/be?t/{print$0}’
bet
$echo“beet”|gawk‘/be?t/{print$0}’
$
$echo“beeet”|gawk‘/be?t/{print$0}’
$
Iftheecharacterdoesn’tappearinthetext,oraslongasitappearsonlyonceinthetext,
thepatternmatches.
Aswiththeasterisk,youcanusethequestionmarksymbolalongwithacharacterclass:
$echo“bt”|gawk‘/b[ae]?t/{print$0}’
bt
$echo“bat”|gawk‘/b[ae]?t/{print$0}’
bat
$echo“bot”|gawk‘/b[ae]?t/{print$0}’
$
$echo“bet”|gawk‘/b[ae]?t/{print$0}’
bet
$echo“baet”|gawk‘/b[ae]?t/{print$0}’
$
$echo“beat”|gawk‘/b[ae]?t/{print$0}’
$
$echo“beet”|gawk‘/b[ae]?t/{print$0}’
$
If zero or one character from the character class appears, the pattern match passes.
However,ifbothcharactersappear,orifoneofthecharactersappearstwice,thepattern
matchfails.
Theplussign
Theplussignisanotherpatternsymbolthat’ssimilartotheasterisk,butwithadifferent
twist than the question mark. The plus sign indicates that the preceding character can
appearoneormoretimes,butmustbepresentatleastonce.Thepatterndoesn’tmatchif
thecharacterisnotpresent:
$echo“beeet”|gawk‘/be+t/{print$0}’
beeet
$echo“beet”|gawk‘/be+t/{print$0}’
beet
$echo“bet”|gawk‘/be+t/{print$0}’
bet
$echo“bt”|gawk‘/be+t/{print$0}’
$
If the e character is not present, the pattern match fails. The plus sign also works with
characterclasses,thesamewayastheasteriskandquestionmarkdo:
$echo“bt”|gawk‘/b[ae]+t/{print$0}’
$
$echo“bat”|gawk‘/b[ae]+t/{print$0}’
bat
$echo“bet”|gawk‘/b[ae]+t/{print$0}’
bet
$echo“beat”|gawk‘/b[ae]+t/{print$0}’
beat
$echo“beet”|gawk‘/b[ae]+t/{print$0}’
beet
$echo“beeat”|gawk‘/b[ae]+t/{print$0}’
beeat
$
This time if either character defined in the character class appears, the text matches the
specifiedpattern.
Usingbraces
CurlybracesareavailableinEREtoallowyoutospecifyalimitonarepeatableregular
expression. This is often referred to as an interval. You can express the interval in two
formats:
m:Theregularexpressionappearsexactlymtimes.
m,n:Theregularexpressionappearsatleastmtimes,butnomorethanntimes.
This feature allows you to fine-tune exactly how many times you allow a character (or
characterclass)toappearinapattern.
Caution
Bydefault,thegawkprogramdoesn’trecognizeregularexpressionintervals.You
mustspecifythe—re-intervalcommandlineoptionforthegawkprogramto
recognizeregularexpressionintervals.
Here’sanexampleofusingasimpleintervalofonevalue:
$echo“bt”|gawk—re-interval‘/be{1}t/{print$0}’
$
$echo“bet”|gawk—re-interval‘/be{1}t/{print$0}’
bet
$echo“beet”|gawk—re-interval‘/be{1}t/{print$0}’
$
By specifying an interval of one, you restrict the number of times the character can be
presentforthestringtomatchthepattern.Ifthecharacterappearsmoretimes,thepattern
matchfails.
Often,specifyingthelowerandupperlimitcomesinhandy:
$echo“bt”|gawk—re-interval‘/be{1,2}t/{print$0}’
$
$echo“bet”|gawk—re-interval‘/be{1,2}t/{print$0}’
bet
$echo“beet”|gawk—re-interval‘/be{1,2}t/{print$0}’
beet
$echo“beeet”|gawk—re-interval‘/be{1,2}t/{print$0}’
$
Inthisexample,the e character can appear once or twice for the pattern match to pass;
otherwise,thepatternmatchfails.
Theintervalpatternmatchalsoappliestocharacterclasses:
$echo“bt”|gawk—re-interval‘/b[ae]{1,2}t/{print$0}’
$
$echo“bat”|gawk—re-interval‘/b[ae]{1,2}t/{print$0}’
bat
$echo“bet”|gawk—re-interval‘/b[ae]{1,2}t/{print$0}’
bet
$echo“beat”|gawk—re-interval‘/b[ae]{1,2}t/{print$0}’
beat
$echo“beet”|gawk—re-interval‘/b[ae]{1,2}t/{print$0}’
beet
$echo“beeat”|gawk—re-interval‘/b[ae]{1,2}t/{print$0}’
$
$echo“baeet”|gawk—re-interval‘/b[ae]{1,2}t/{print$0}’
$
$echo“baeaet”|gawk—re-interval‘/b[ae]{1,2}t/{print$0}’
$
This regular expression pattern matches if there are exactly one or two instances of the
letteraoreinthetextpattern,butitfailsifthereareanymoreinanycombination.
Thepipesymbol
Thepipesymbolallowstoyoutospecifytwoormorepatternsthattheregularexpression
engineusesinalogicalORformulawhenexaminingthedatastream.Ifanyofthepatterns
matchthedatastreamtext,thetextpasses.Ifnoneofthepatternsmatch,thedatastream
textfails.
Here’stheformatforusingthepipesymbol:
expr1|expr2|…
Here’sanexampleofthis:
$echo“Thecatisasleep”|gawk‘/cat|dog/{print$0}’
Thecatisasleep
$echo“Thedogisasleep”|gawk‘/cat|dog/{print$0}’
Thedogisasleep
$echo“Thesheepisasleep”|gawk‘/cat|dog/{print$0}’
$
This example looks for the regular expression cat or dog in the data stream. You can’t
placeanyspaceswithintheregularexpressionsandthepipesymbol,orthey’readdedto
theregularexpressionpattern.
Theregularexpressionsoneithersideofthepipesymbolcanuseanyregularexpression
pattern,includingcharacterclasses,todefinethetext:
$echo“Hehasahat.”|gawk‘/[ch]at|dog/{print$0}’
Hehasahat.
$
Thisexamplewouldmatchcat,hat,ordoginthedatastreamtext.
Groupingexpressions
Regularexpressionpatternscanalsobegroupedbyusingparentheses.Whenyougroupa
regularexpressionpattern,thegroupistreatedlikeastandardcharacter.Youcanapplya
specialcharactertothegroupjustasyouwouldtoaregularcharacter.Forexample:
$echo“Sat”|gawk‘/Sat(urday)?/{print$0}’
Sat
$echo“Saturday”|gawk‘/Sat(urday)?/{print$0}’
Saturday
$
The grouping of the “urday” ending along with the question mark allows the pattern to
matcheitherthefulldaynameSaturdayortheabbreviatednameSat.
It’s common to use grouping along with the pipe symbol to create groups of possible
patternmatches:
$echo“cat”|gawk‘/(c|b)a(b|t)/{print$0}’
cat
$echo“cab”|gawk‘/(c|b)a(b|t)/{print$0}’
cab
$echo“bat”|gawk‘/(c|b)a(b|t)/{print$0}’
bat
$echo“bab”|gawk‘/(c|b)a(b|t)/{print$0}’
bab
$echo“tab”|gawk‘/(c|b)a(b|t)/{print$0}’
$
$echo“tac”|gawk‘/(c|b)a(b|t)/{print$0}’
$
Thepattern (c|b)a(b|t)matchesanycombinationofthelettersinthefirstgroupalong
withanycombinationofthelettersinthesecondgroup.
RegularExpressionsinAction
Now that you’ve seen the rules and a few simple demonstrations of using regular
expression patterns, it’s time to put that knowledge into action. The following sections
demonstratesomecommonregularexpressionexampleswithinshellscripts.
Countingdirectoryfiles
To start things out, let’s look at a shell script that counts the executable files that are
presentinthedirectoriesdefinedinyour PATHenvironmentvariable.Todothat,youneed
toparseoutthePATHvariableintoseparatedirectorynames.Chapter6showedyouhowto
displaythePATHenvironmentvariable:
$echo$PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/loca
$
Your PATH environment variable will differ, depending on where the applications are
locatedonyourLinuxsystem.Thekeyistorecognizethateachdirectoryinthe PATHis
separatedbyacolon.Togetalistingofdirectoriesthatyoucanuseinascript,youmust
replace each colon with a space. You now recognize that the sededitorcandojustthat
usingasimpleregularexpression:
$echo$PATH|sed‘s/://g’
/usr/local/sbin/usr/local/bin/usr/sbin/usr/bin/sbin/bin
/usr/games/usr/local/games
$
Afteryouhavethedirectoriesseparatedout,youcanusetheminastandardforstatement
(seeChapter13)toiteratethrougheachdirectory:
mypath=$(echo$PATH|sed‘s/://g’)
fordirectoryin$mypath
do
…
done
After you have each directory, you can use the ls command to list each file in each
directory, and use another for statement to iterate through each file, incrementing a
counterforeachfile.
Thefinalversionofthescriptlookslikethis:
$catcountfiles
#!/bin/bash
#countnumberoffilesinyourPATH
mypath=$(echo$PATH|sed‘s/://g’)
count=0
fordirectoryin$mypath
do
check=$(ls$directory)
foritemin$check
do
count=$[$count+1]
done
echo“$directory-$count”
count=0
done
$./countfiles/usr/local/sbin-0
/usr/local/bin-2
/usr/sbin-213
/usr/bin-1427
/sbin-186
/bin-152
/usr/games-5
/usr/local/games–0
$
Nowwe’restartingtoseesomeofthepowerbehindregularexpressions!
Validatingaphonenumber
The previous example showed how to incorporate the simple regular expression along
withsedtoreplacecharactersinadatastreamtoprocessdata.Often,regularexpressions
areusedtovalidatedatatoensurethatdataisinthecorrectformatforascript.
A common data validation application checks phone numbers. Often, data entry forms
request phone numbers, and often customers fail to enter a properly formatted phone
number.PeopleintheUnitedStatesuseseveralcommonwaystodisplayaphonenumber:
(123)456-7890
(123)456-7890
123-456-7890
123.456.7890
Thisleavesfourpossibilitiesforhowcustomerscanentertheirphonenumberinaform.
Theregularexpressionmustberobustenoughtohandleanyofthesesituations.
Whenbuildingaregularexpression,it’sbesttostartontheleftsideandbuildyourpattern
tomatchthepossiblecharactersyou’llruninto.Inthisexample,theremayormaynotbe
aleftparenthesisinthephonenumber.Thiscanbematchedbyusingthepattern:
^\(?
The caret is used to indicate the beginning of the data. Because the left parenthesis is a
special character, you must escape it to use it as a normal character. The question mark
indicatesthattheleftparenthesismayormaynotappearinthedatatomatch.
Nextisthethree-digitareacode.IntheUnitedStates,areacodesstartwiththenumber2
(noareacodesstartwiththedigits0or1),andcangoto9.Tomatchtheareacode,you’d
usethefollowingpattern:
[2-9][0-9]{2}
This requires that the first character be a digit between 2 and 9, followed by any two
digits.Aftertheareacode,theendingrightparenthesismayormaynotappear:
\)?
Aftertheareacode,therecanbeaspace,nospace,adash,oradot.Youcangroupthose
usingacharactergroupalongwiththepipesymbol:
(||-|.)
Theveryfirstpipesymbolappearsimmediatelyaftertheleftparenthesistomatchtheno
spacecondition.Youmustusetheescapecharacterforthedot;otherwise,itisinterpreted
tomatchanycharacter.
Nextisthethree-digitphoneexchangenumber.Nothingspecialisrequiredhere:
[0-9]{3}
Afterthephoneexchangenumber,youmustmatchaspace,adash,oradot(thistimeyou
don’t have to worry about matching no space because there must be at least a space
betweenthephoneexchangenumberandtherestofthenumber):
(|-|.)
Thentofinishthingsoff,youmustmatchthefour-digitlocalphoneextensionattheendof
thestring:
[0-9]{4}$
Puttingtheentirepatterntogetherresultsinthis:
^\(?[2-9][0-9]{2}\)?(||-|.)[0-9]{3}(|-|.)[0-9]{4}$
You can use this regular expression pattern in the gawk program to filter out bad phone
numbers. Now you just need to create a simple script using the regular expression in a
gawkprogramandfilteryourphonelistthroughthescript.Rememberthatwhenyouuse
regular expression intervals in the gawk program, you must use the —re-interval
commandlineoption,oryouwon’tgetthecorrectresults.
Here’sthescript:
$catisphone
#!/bin/bash
#scripttofilteroutbadphonenumbers
gawk—re-interval‘/^\(?[2-9][0-9]{2}\)?(||-|\┐
[0-9]{3}(|-|.)[0-9]{4}/{print$0}’
$
Althoughyoucan’ttellfromthislisting,thegawkcommandisonasinglelineintheshell
script.Youcanthenredirectphonenumberstothescriptforprocessing:
$echo“317-555-1234”|./isphone
317-555-1234
$echo“000-555-1234”|./isphone
$echo“312555-1234”|./isphone
312555-1234
$
Oryoucanredirectanentirefileofphonenumberstofilterouttheinvalidones:
$catphonelist
000-000-0000
123-456-7890
212-555-1234
(317)555-1234
(202)555-9876
33523
1234567890
234.123.4567
$catphonelist|./isphone
212-555-1234
(317)555-1234
(202)555-9876
234.123.4567
$
Onlythevalidphonenumbersthatmatchtheregularexpressionpatternappear.
Parsingane-mailaddress
Thesedays,e-mailhasbecomeacrucialformofcommunication.Tryingtovalidatee-mail
addresseshasbecomequiteachallengeforscriptbuildersbecauseofthemyriadwaysto
createane-mailaddress.Thisisthebasicformofane-mailaddress:
username@hostname
The username value can use any alphanumeric character, along with several special
characters:
Dot
Dash
Plussign
Underscore
These characters can appear in any combination in a valid e-mail userid. The hostname
portionofthee-mailaddressconsistsofoneormoredomainnamesandaservername.
The server and domain names must also follow strict naming rules, allowing only
alphanumericcharacters,alongwiththespecialcharacters:
Dot
Underscore
Theserveranddomainnamesareeachseparatedbyadot,withtheservernamespecified
first,anysubdomainnamesspecifiednext,andfinally,thetop-leveldomainnamewithout
atrailingdot.
At one time, the top-level domains were fairly limited, and regular expression pattern
buildersattemptedtoaddthemallinpatternsforvalidation.Unfortunately,astheInternet
grew,sodidthepossibletop-leveldomains.Thistechniqueisnolongeraviablesolution.
Let’sstartbuildingtheregularexpressionpatternfromtheleftside.Weknowthatthere
canbemultiplevalidcharactersintheusername.Thisshouldbefairlyeasy:
^([a-zA-Z0-9_-.\+]+)@
This grouping specifies the allowable characters in the username and the plus sign to
indicatethatatleastonecharactermustbepresent.Thenextcharacterobviouslyisthe@
symbol—nosurprisesthere.
The hostname pattern uses the same technique to match the server name and the
subdomainnames:
([a-zA-Z0-9_-.]+)
Thispatternmatchesthetext:
server
server.subdomain
server.subdomain.subdomain
There are special rules for the top-level domain. Top-level domains are only alphabetic
characters,andtheymustbenofewerthantwocharacters(usedincountrycodes)andno
morethanfivecharactersinlength.Thefollowingistheregularexpressionpatternforthe
top-leveldomain:
.([a-zA-Z]{2,5})$
Puttingtheentirepatterntogetherresultsinthefollowing:
^([a-zA-Z0-9_-.\+]+)@([a-zA-Z0-9_-.]+).([a-zA-Z]{2,5})$
This pattern filters out poorly formatted e-mail addresses from a data list. Now you can
createyourscripttoimplementtheregularexpression:
$echo“[email protected]”|./isemail
[email protected]
$echo“[email protected].”|./isemail
$
$echo“[email protected]”|./isemail
$
$echo“rich@here-now”|./isemail
$
$echo“[email protected]”|./isemail
[email protected]
$echo“[email protected]”|./isemail
[email protected]
$echo“rich/[email protected]”|./isemail
$
$echo“rich#[email protected]”|./isemail
$
$echo“rich*[email protected]”|./isemail
$
Summary
If you manipulate data files in shell scripts, you need to become familiar with regular
expressions. Regular expressions are implemented in Linux utilities, programming
languages,andapplicationsusingregularexpressionengines.Ahostofdifferentregular
expressionenginesisavailableintheLinuxworld.ThetwomostpopulararethePOSIX
Basic Regular Expression (BRE) engine and the POSIX Extended Regular Expression
(ERE) engine. The sed editor conforms mainly to the BRE engine, while the gawk
programutilizesmostfeaturesfoundintheEREengine.
Aregularexpressiondefinesapatterntemplatethat’susedtofiltertextinadatastream.
The pattern consists of a combination of standard text characters and special characters.
Thespecialcharactersareusedbytheregularexpressionenginetomatchaseriesofone
ormorecharacters,similarlytohowwildcardcharactersworkinotherapplications.
Bycombiningcharactersandspecialcharacters,youcandefineapatterntomatchalmost
anytypeofdata.Youcanthenusethe sededitoror gawkprogramtofilterspecificdata
fromalargerdatastream,orforvalidatingdatareceivedfromdataentryapplications.
The next chapter digs deeper into using the sed editor to perform advanced text
manipulation.Lotsofadvancedfeaturesareavailableinthe sededitorthatmakeituseful
forhandlinglargedatastreamsandfilteringoutjustwhatyouneed.
Chapter21
Advancedsed
InThisChapter
1. Usingmultilinecommands
2. Understandingtheholdspace
3. Negatingacommand
4. Changingtheflow
5. Replacingviaapattern
6. Usingsedinscripts
7. Creatingsedutilities
Chapter19showedyouhowtousethebasicsofthesededitortomanipulatetextin
datastreams.Thebasicsededitorcommandsarecapableofhandlingmostofyour
everydaytext-editingrequirements.Thischapterlooksatthemoreadvancedfeatures
thatthesededitorhastooffer.Thesearefeaturesthatyoumightnotuseasoften.But
whenyouneedthem,it’snicetoknowthatthey’reavailableandhowtousethem.
LookingatMultilineCommands
Whenusingthebasic sed editor commands, you may have noticed a limitation. All the
sededitorcommandsperformfunctionsonasinglelineofdata.Asthe sededitorreadsa
datastream,itdividesthedataintolinesbasedonthepresenceofnewlinecharacters.The
sededitorhandleseachdatalineoneatatime,processingthedefinedscriptcommandson
thedataline,andthenmovingontothenextlineandrepeatingtheprocessing.
Sometimes, you need to perform actions on data that spans more than one line. This is
especiallytrueifyou’retryingtofindorreplaceaphrase.
Forexample,ifyou’relookingforthephrase Linux System Administrators Groupin
your data, it’s quite possible that the phrase’s words can be split onto two lines. If you
processedthetextusinganormal sededitorcommand,itwouldbeimpossibletodetect
thesplitphrase.
Fortunately, the designers behind the sed editor thought of that situation and devised a
solution. The sed editor includes three special commands that you can use to process
multilinetext:
Naddsthenextlineinthedatastreamtocreateamultilinegroupforprocessing.
Ddeletesasinglelineinamultilinegroup.
Pprintsasinglelineinamultilinegroup.
Thefollowingsectionsexaminethesemultilinecommandsmorecloselyanddemonstrate
howyoucanusetheminyourscripts.
Navigatingthenextcommand
Beforeyoucanexaminethemultiline nextcommand,youfirstneedtolookathowthe
single-lineversionofthenextcommandworks.Afteryouknowwhatthatcommanddoes,
it’smucheasiertounderstandhowthemultilineversionofthenextcommandoperates.
Usingthesingle-linenextcommand
Thelowercase ncommandtellsthe sededitortomovetothenextlineoftextinthedata
stream,withoutgoingbacktothebeginningofthecommands.Rememberthatnormally
thesededitorprocessesallthedefinedcommandsonalinebeforemovingtothenextline
oftextinthedatastream.Thesingle-linenextcommandaltersthisflow.
Thismaysoundsomewhatcomplicated,andsometimesitis.Inthisexample,youhavea
datafilethatcontainsfivelines,twoofwhichareempty.Thegoalistoremovetheblank
lineaftertheheaderlinebutleavetheblanklinebeforethelastlineintact.Ifyouwritea
sedscripttojustremoveblanklines,youremovebothblanklines:
$catdata1.txt
Thisistheheaderline.
Thisisadataline.
Thisisthelastline.
$
$sed‘/∧$/d’data1.txt
Thisistheheaderline.
Thisisadataline.
Thisisthelastline.
$
Becausethelineyouwanttoremoveisblank,youdon’thaveanytextyoucansearchfor
touniquelyidentifytheline.Thesolutionistousethe ncommand.Inthisnextexample,
thescriptlooksforauniquelinethatcontainstheword header.Afterthescriptidentifies
thatline,the ncommandmovesthe sededitortothenextlineoftext,whichistheempty
line.
$sed‘/header/{n;d}’data1.txt
Thisistheheaderline.
Thisisadataline.
Thisisthelastline.
$
At that point, the sed editor continues processing the command list, which uses the d
commandtodeletetheemptyline.Whenthe sededitorreachestheendofthecommand
script,itreadsthenextlineoftextfromthedatastreamandstartsprocessingcommands
from the top of the command script. The sed editor does not find another line with the
wordheader;thus,nofurtherlinesaredeleted.
Combininglinesoftext
Now that you’ve seen the single-line next command, you can look at the multiline
version.Thesingle-line nextcommandmovesthenextlineoftextfromthedatastream
into the processing space (called the pattern space) of the sed editor. The multiline
versionofthenextcommand(whichusesacapitalN)addsthenextlineoftexttothetext
alreadyinthepatternspace.
This has the effect of combining two lines of text from the data stream into the same
patternspace.Thelinesoftextarestillseparatedbyanewlinecharacter,butthesededitor
cannowtreatbothlinesoftextasoneline.
Here’sademonstrationofhowtheNcommandoperates:
$catdata2.txt
Thisistheheaderline.
Thisisthefirstdataline.
Thisistheseconddataline.
Thisisthelastline.
$
$sed‘/first/{N;s/\n//}’data2.txt
Thisistheheaderline.
Thisisthefirstdataline.Thisistheseconddataline.
Thisisthelastline.
$
Thesededitorscriptsearchesforthelineoftextthatcontainstheword“first”init.When
itfindstheline,itusestheNcommandtocombinethenextlinewiththatline.Itthenuses
the substitutioncommand(s)toreplacethenewlinecharacterwithaspace.Theresult
isthatthetwolinesinthetextfileappearasonelineinthesededitoroutput.
This has a practical application if you’re searching for a text phrase that may be split
betweentwolinesinthedatafile.Here’sanexample:
$catdata3.txt
OnTuesday,theLinuxSystem
Administrator’sgroupmeetingwillbeheld.
AllSystemAdministratorsshouldattend.
Thankyouforyourattendance.
$
$sed‘N;s/SystemAdministrator/DesktopUser/’data3.txt
OnTuesday,theLinuxSystem
Administrator’sgroupmeetingwillbeheld.
AllDesktopUsersshouldattend.
Thankyouforyourattendance.
$
The substitution command is looking for the specific two-word phrase System
Administratorinthetextfile.Inthesinglelinewherethephraseappears,everythingis
fine; the substitution command can replace the text. But in the situation where the
phrase is split between two lines, the substitution command doesn’t recognize the
matchingpattern.
TheNcommandhelpssolvethisproblem:
$sed‘N;s/System.Administrator/DesktopUser/’data3.txt
OnTuesday,theLinuxDesktopUser’sgroupmeetingwillbeheld.
AllDesktopUsersshouldattend.
Thankyouforyourattendance.
$
Byusingthe N command to combine the next line with the line where the first word is
found,youcandetectwhenalinesplitoccursinthephrase.
Notice that the substitution command uses a wildcard pattern (.) between the word
Systemandtheword Administratortomatchboththespaceandthenewlinesituation.
However,whenitmatchedthenewlinecharacter,itremoveditfromthestring,causingthe
twolinestomergeintooneline.Thismaynotbeexactlywhatyouwant.
Tosolvethisproblem,youcanusetwo substitutioncommandsinthe sededitorscript,
onetomatchthemultilineoccurrenceandonetomatchthesingle-lineoccurrence:
$sed‘N
>s/System\nAdministrator/Desktop\nUser/
>s/SystemAdministrator/DesktopUser/
>‘data3.txt
OnTuesday,theLinuxDesktop
User’sgroupmeetingwillbeheld.
AllDesktopUsersshouldattend.
Thankyouforyourattendance.
$
Thefirstsubstitutioncommandspecificallylooksforthenewlinecharacterbetweenthe
two search words and includes it in the replacement string. This allows you to add the
newlinecharacterinthesameplaceinthenewtext.
There’sstillonesubtleproblemwiththisscript,however.Thescriptalwaysreadsthenext
line of text into the pattern space before executing the sed editor commands. When it
reachesthelastlineoftext,thereisn’tanextlineoftexttoread,sotheNcommandcauses
the sed editor to stop. If the matching text is on the last line in the data stream, the
commandsdon’tcatchthematchingdata:
$catdata4.txt
OnTuesday,theLinuxSystem
Administrator’sgroupmeetingwillbeheld.
AllSystemAdministratorsshouldattend.
$
$sed‘N
>s/System\nAdministrator/Desktop\nUser/
>s/SystemAdministrator/DesktopUser/
>‘data4.txt
OnTuesday,theLinuxDesktop
User’sgroupmeetingwillbeheld.
AllSystemAdministratorsshouldattend.
$
Becausethe SystemAdministratortextappearsinthelastlineinthedatastream,the N
commandmissesit,asthereisn’tanotherlinetoreadintothepatternspacetocombine.
You can easily resolve this problem by moving your single-line commands before the N
commandandhavingonlythemultilinecommandsappearaftertheNcommand,likethis:
$sed‘
>s/SystemAdministrator/DesktopUser/
>N
>s/System\nAdministrator/Desktop\nUser/
>‘data4.txt
OnTuesday,theLinuxDesktop
User’sgroupmeetingwillbeheld.
AllDesktopUsersshouldattend.
$
Now,thesubstitutioncommandthatlooksforthephraseinasinglelineworksjustfine
onthelastlineinthedatastream,andthemultiline substitutioncommandcoversthe
occurrenceinthemiddleofthedatastream.
Navigatingthemultilinedeletecommand
Chapter 19 introduced the single-line delete command (d). The sed editor uses it to
delete the current line in the pattern space. If you’re working with the N command,
however,youmustbecarefulwhenusingthesingle-linedeletecommand:
$sed‘N;/System\nAdministrator/d’data4.txt
AllSystemAdministratorsshouldattend.
$
The deletecommandlookedforthewords Systemand Administratorinseparatelines
anddeletedbothofthelinesinthepatternspace.Thismayormaynothavebeenwhatyou
intended.
The sed editor provides the multiline deletecommand(D), which deletes only the first
line in the pattern space. It removes all characters up to and including the newline
character:
$sed‘N;/System\nAdministrator/D’data4.txt
Administrator’sgroupmeetingwillbeheld.
AllSystemAdministratorsshouldattend.
$
Thesecondlineoftext,addedtothepatternspacebytheNcommand,remainsintact.This
comesinhandyifyouneedtoremovealineoftextthatappearsbeforealinethatyoufind
adatastringin.
Here’s an example of removing a blank line that appears before the first line in a data
stream:
$catdata5.txt
Thisistheheaderline.
Thisisadataline.
Thisisthelastline.
$
$sed‘/∧$/{N;/header/D}’data5.txt
Thisistheheaderline.
Thisisadataline.
Thisisthelastline.
$
This sededitorscriptlooksforblanklinesandthenusesthe Ncommandtoaddthenext
line of text into the pattern space. If the new pattern space contents contain the word
header, the D command removes the first line in the pattern space. Without the
combinationofthe Nand Dcommands,itwouldbeimpossibletoremovethefirstblank
linewithoutremovingallotherblanklines.
Navigatingthemultilineprintcommand
By now, you’re probably catching on to the difference between the single-line and
multilineversionsofthecommands.Themultilineprintcommand(P)followsalongusing
thesametechnique.Itprintsonlythefirstlineinamultilinepatternspace.Thisincludes
allcharactersuptothenewlinecharacterinthepatternspace.Itisusedinmuchthesame
wayasthesingle-line pcommandtodisplaytextwhenyouusethe -noptiontosuppress
outputfromthescript.
$sed-n‘N;/System\nAdministrator/P’data3.txt
OnTuesday,theLinuxSystem
$
Whenthemultilinematchoccurs,the Pcommandprintsonlythefirstlineinthepattern
space.Thepowerofthemultiline Pcommandcomesintoplaywhenyoucombineitwith
theNandDmultilinecommands.
The D command has a unique feature in that it forces the sed editor to return to the
beginningofthescriptandrepeatthecommandsonthesamepatternspace(itdoesn’tread
a new line of text from the data stream). By including the N command in the command
script,youcaneffectivelysingle-stepthroughthepatternspace,matchingmultiplelines
together.
Next,byusingthePcommand,youcanprintthefirstline,andthenusingtheDcommand,
you can delete the first line and loop back to the beginning of the script. When you are
backatthescript’sbeginning,the Ncommandreadsinthenextlineoftextandstartsthe
processalloveragain.Thisloopcontinuesuntilyoureachtheendofthedatastream.
HoldingSpace
Thepatternspaceisanactivebufferareathatholdsthetextexaminedbythe sededitor
whileitprocessescommands.However,itisn’ttheonlyspaceavailableinthe sededitor
forstoringtext.
The sededitorutilizesanotherbufferareacalledthe hold space. You can use the hold
spacetotemporarilyholdlinesoftextwhileworkingonotherlinesinthepatternspace.
The five commands associated with operating with the hold space are shown in Table
21.1.
Table21.1ThesedEditorHoldSpaceCommands
Command
h
H
g
G
x
Description
Copiespatternspacetoholdspace
Appendspatternspacetoholdspace
Copiesholdspacetopatternspace
Appendsholdspacetopatternspace
Exchangescontentsofpatternandholdspaces
Thesecommandsletyoucopytextfromthepatternspacetotheholdspace.Thisfreesup
thepatternspacetoloadanotherstringforprocessing.
Usually,afterusingthe hor Hcommandstomoveastringtotheholdspace,eventually
youwanttousethe g, G,or xcommandstomovethestoredstringbackintothepattern
space(otherwise,youwouldn’thavecaredaboutsavingtheminthefirstplace).
With two buffer areas, trying to determine what line of text is in which buffer area can
sometimesgetconfusing.Here’sashortexamplethatdemonstrateshowtousethehandg
commandstomovedatabackandforthbetweenthesededitorbufferspaces:
$catdata2.txt
Thisistheheaderline.
Thisisthefirstdataline.
Thisistheseconddataline.
Thisisthelastline.
$
$sed-n‘/first/{h;p;n;p;g;p}’data2.txt
Thisisthefirstdataline.
Thisistheseconddataline.
Thisisthefirstdataline.
$
Lookattheprecedingcodeexamplestepbystep:
1. Thesedscriptusesaregularexpressionintheaddresstofilterthelinecontainingthe
wordfirst.
2. Whenthelinecontainingthewordfirstappears,theinitialcommandin{},theh
command,placesthelineintheholdspace.
3. Thenextcommand,thepcommand,printsthecontentsofthepatternspace,whichis
stillthefirstdataline.
4. Thencommandretrievesthenextlineinthedatastream(Thisistheseconddata
line)andplacesitinthepatternspace.
5. Thepcommandprintsthecontentsofthepatternspace,whichisnowtheseconddata
line.
6. Thegcommandplacesthecontentsoftheholdspace(Thisisthefirstdata
line)backintothepatternspace,replacingthecurrenttext.
7. Thepcommandprintsthecurrentcontentsofthepatternspace,whichisnowbackto
thefirstdataline.
Byshufflingthetextlinesaroundusingtheholdspace,youcanforcethefirstdatalineto
appearaftertheseconddatalineintheoutput.Ifyoujustdropthefirst pcommand,you
canoutputthetwolinesinreverseorder:
$sed-n‘/first/{h;n;p;g;p}’data2.txt
Thisistheseconddataline.
Thisisthefirstdataline.
$
Thisisthestartofsomethinguseful.Youcanusethistechniquetocreatea sedscriptthat
reverses an entire file of text lines! To do that, however, you need to see the negating
featureofthesededitor,whichiswhatthenextsectionisallabout.
NegatingaCommand
Chapter19showedthatthe sededitorappliescommandseithertoeverytextlineinthe
datastreamortolinesspecificallyindicatedbyeitherasingleaddressoranaddressrange.
Youcanalsoconfigureacommandtonotapplytoaspecificaddressoraddressrangein
thedatastream.
The exclamation mark command (!) is used to negate a command. This means in
situations where the command would normally have been activated, it isn’t. Here’s an
exampledemonstratingthisfeature:
$sed-n‘/header/!p’data2.txt
Thisisthefirstdataline.
Thisistheseconddataline.
Thisisthelastline.
$
Thenormal pcommandwouldhaveprintedonlythelineinthe data2filethatcontained
thewordheader.Byaddingtheexclamationmark,theoppositehappens—alllinesinthe
fileareprintedexcepttheonethatcontainedthewordheader.
Usingtheexclamationmarkcomesinhandyinseveralapplications.Recallthatearlierin
thechapter,the“Navigatingthenextcommand”sectionshowedasituationwherea sed
editorcommandwouldn’toperateonthelastlineoftextinthedatastreambecausethere
wasn’talineafterit.Youcanusetheexclamationpointtofixthatproblem:
$sed‘N;
>s/System\nAdministrator/Desktop\nUser/
>s/SystemAdministrator/DesktopUser/
>‘data4.txt
OnTuesday,theLinuxDesktop
User’sgroupmeetingwillbeheld.
AllSystemAdministratorsshouldattend.
$
$sed‘$!N;
>s/System\nAdministrator/Desktop\nUser/
>s/SystemAdministrator/DesktopUser/
>‘data4.txt
OnTuesday,theLinuxDesktop
User’sgroupmeetingwillbeheld.
AllDesktopUsersshouldattend.
$
This example shows the exclamation mark used with the N command, along with the
dollarsign($)specialaddress.Thedollarsignrepresentsthelastlineoftextinthedata
stream, so when the sed editor reaches the last line, it doesn’t execute the N command.
However,forallotherlines,itdoesexecutethecommand.
Usingthistechnique,youcanreversetheorderoftextlinesinadatastream.Toreverse
theorderofthelinesastheyappearinthetextstream(displaythelastlinefirstandthe
firstlinelast),youneedtodosomefancyfootworkusingtheholdspace.
Thepatternyouneedtoworkwithgoeslikethis:
1. Placealineinthepatternspace.
2. Placethelinefromthepatternspacetotheholdspace.
3. Putthenextlineoftextinthepatternspace.
4. Appendtheholdspacetothepatternspace.
5. Placeeverythinginthepatternspaceintotheholdspace.
6. RepeatSteps3through5untilyou’veputallthelinesinreverseorderinthehold
space.
7. Retrievethelines,andprintthem.
Figure21.1diagramswhatthislookslikeinmoredetail.
Figure21.1Reversingtheorderofatextfileusingtheholdspace
When using this technique, you do not want to print lines as they are processed. This
meansusingthe -ncommandlineoptionfor sed.Thenextthingtodetermineishowto
appendtheholdspacetexttothepatternspacetext.ThisisdonebyusingtheGcommand.
Theonlyproblemisthatyoudon’twanttoappendtheholdspacetothefirstlineoftext
processed.Thisiseasilysolvedbyusingtheexclamationmarkcommand:
1!G
The next step is to place the new pattern space (the text line with the appended reverse
lines)intotheholdspace.Thisissimpleenough;justusethehcommand.
Whenyou’vegottheentiredatastreaminthepatternspaceinreverseorder,youjustneed
toprinttheresults.Youknowyouhavetheentiredatastreaminthepatternspacewhen
you’vereachedthelastlineinthedatastream.Toprinttheresults,justusethefollowing
command:
$p
Thosearethepiecesyouneedtocreateyourline-reversing sededitorscript.Nowtryit
outinatestrun:
$catdata2.txt
Thisistheheaderline.
Thisisthefirstdataline.
Thisistheseconddataline.
Thisisthelastline.
$
$sed-n‘{1!G;h;$p}’data2.txt
Thisisthelastline.
Thisistheseconddataline.
Thisisthefirstdataline.
Thisistheheaderline.
$
The sed editor script performed as expected. The output from the script reverses the
originallinesinthetextfile.Thisdemonstratesthepowerofusingtheholdspaceinyour
sedscripts.Itprovidesaneasywaytomanipulatetheorderoflinesinthescriptoutput.
Note
Incaseyou’rewondering,abashshellcommandcanperformthefunctionof
reversingatextfile.Thetaccommanddisplaysatextfileinreverseorder.You
probablynoticedtheclevernameofthecommandbecauseitperformsthereverse
functionofthecatcommand.
ChangingtheFlow
Normally,the sed editor processes commands starting at the top and proceeding toward
theendofthescript(theexceptionistheDcommand,whichforcesthesededitortoreturn
to the top of the script without reading a new line of text). The sed editor provides a
methodforalteringtheflowofthecommandscript,producingaresultsimilartothatofa
structuredprogrammingenvironment.
Branching
Intheprevioussection,yousawhowtheexclamationmarkcommandisusedtonegatethe
effectofacommandonalineoftext.The sededitorprovidesawaytonegateanentire
sectionofcommands,basedonanaddress,anaddresspattern,oranaddressrange.This
allows you to perform a group of commands only on a specific subset within the data
stream.
Here’stheformatofthebranchcommand:
[address]b[label]
The address parameter determines which line or lines of data trigger the branch
command.The labelparameterdefinesthelocationtobranchto.Ifthe labelparameter
isnotpresent,thebranchcommandproceedstotheendofthescript.
$catdata2.txt
Thisistheheaderline.
Thisisthefirstdataline.
Thisistheseconddataline.
Thisisthelastline.
$
$sed‘{2,3b;s/Thisis/Isthis/;s/line./test?/}’data2.txt
Isthistheheadertest?
Thisisthefirstdataline.
Thisistheseconddataline.
Isthisthelasttest?
$
The branch command skips the two substitution commands for the second and third
linesinthedatastream.
Insteadofgoingtotheendofthescript,youcandefinealabelforthebranchcommandto
jumpto.Labelsstartwithacolonandcanbeuptosevencharactersinlength:
:label2
To specify the label, just add it after the b command. Using labels allows you to skip
commandsthatmatchthebranchaddressbutstillprocessothercommandsinthescript:
$sed‘{/first/bjump1;s/Thisisthe/Nojumpon/
>:jump1
>s/Thisisthe/Jumphereon/}’data2.txt
Nojumponheaderline
Jumphereonfirstdataline
Nojumponseconddataline
Nojumponlastline
$
The branch command specifies that the program should jump to the script line labeled
jump1 if the matching text “first” appears in the line. If the branch command pattern
doesn’tmatch,the sededitorcontinuesprocessingcommandsinthescript,includingthe
commandafterthe branchlabel.(Thus,allthree substitutioncommandsareprocessed
onlinesthatdon’tmatchthebranchpattern.)
If a line matches the branch pattern, the sed editor branches to the branch label line.
Thus,onlythelastsubstitutioncommandisexecuted.
The example shows branching to a label further down in the sed script. You can also
branchtoalabelthatappearsearlierinthescript,thuscreatingaloopingeffect:
$echo“This,is,a,test,to,remove,commas.”|sed-n‘{
>:start
>s/,//1p
>bstart
>}’
Thisis,a,test,to,remove,commas.
Thisisa,test,to,remove,commas.
Thisisatest,to,remove,commas.
Thisisatestto,remove,commas.
Thisisatesttoremove,commas.
Thisisatesttoremovecommas.
^C
$
Each script iteration removes the first occurrence of a comma from the text string and
printsthestring.There’sonecatchtothisscript:Itneverends.Thissituationcreatesan
endlessloop,searchingforcommasuntilyoumanuallystopitbysendingasignalwiththe
Ctrl+Ckeycombination.
Topreventthisproblem,youshouldspecifyanaddresspatternforthebranchcommandto
lookfor.Ifthepatternisn’tpresent,thebranchingshouldstop:
$echo“This,is,a,test,to,remove,commas.”|sed-n‘{
>:start
>s/,//1p
>/,/bstart
>}’
Thisis,a,test,to,remove,commas.
Thisisa,test,to,remove,commas.
Thisisatest,to,remove,commas.
Thisisatestto,remove,commas.
Thisisatesttoremove,commas.
Thisisatesttoremovecommas.
$
Now the branch command branches only if there’s a comma in the line. After the last
comma has been removed, the branch command doesn’t execute, allowing the script to
properlyfinish.
Testing
Similartothebranchcommand,thetestcommand(t)isalsousedtomodifytheflowof
thesededitorscript.Insteadofjumpingtoalabelbasedonanaddress,thetestcommand
jumpstoalabelbasedontheoutcomeofasubstitutioncommand.
If the substitution command successfully matches and substitutes a pattern, the test
command branches to the specified label. If the substitution command doesn’t match
thespecifiedpattern,thetestcommanddoesn’tbranch.
Thetestcommandusesthesameformatasthebranchcommand:
[address]t[label]
Like the branch command, if you don’t specify a label, sed branches to the end of the
scriptifthetestsucceeds.
The test command provides a cheap way to perform a basic if-then statement on the
textinthedatastream.Forexample,ifyoudon’tneedtomakeasubstitutionifanother
substitutionwasmade,thetestcommandcanhelp:
$sed‘{
>s/first/matched/
>t
>s/Thisisthe/Nomatchon/
>}’data2.txt
Nomatchonheaderline
Thisisthematcheddataline
Nomatchonseconddataline
Nomatchonlastline
$
Thefirstsubstitutioncommandlooksforthepatterntextfirst.Ifitmatchesthepattern
in the line, it replaces the text, and the test command jumps over the second
substitutioncommand.Ifthefirst substitution command doesn’t match the pattern,
thesecondsubstitutioncommandisprocessed.
Usingthetestcommand,youcancleanuptheloopyoutriedusingthebranchcommand:
$echo“This,is,a,test,to,remove,commas.”|sed-n‘{
>:start
>s/,//1p
>tstart
>}’
Thisis,a,test,to,remove,commas.
Thisisa,test,to,remove,commas.
Thisisatest,to,remove,commas.
Thisisatestto,remove,commas.
Thisisatesttoremove,commas.
Thisisatesttoremovecommas.
$
When there are no more substitutions to make, the test command doesn’t branch and
continueswiththerestofthescript.
ReplacingviaaPattern
You’veseenhowtousepatternsinthe sedcommandstoreplacetextinthedatastream.
However, when using wildcard characters it’s not easy to know exactly what text will
matchthepattern.
Forexample,saythatyouwanttoplacedoublequotationmarksaroundawordyoumatch
inaline.That’ssimpleenoughifyou’rejustlookingforonewordinthepatterntomatch:
$echo“Thecatsleepsinhishat.”|sed‘s/cat/“cat”/’
The“cat”sleepsinhishat.
$
Butwhatifyouuseawildcardcharacter(.)inthepatterntomatchmorethanoneword?
$echo“Thecatsleepsinhishat.”|sed‘s/.at/”.at”/g’
The“.at”sleepsinhis“.at”.
$
Thesubstitutionstringusedthedotwildcardcharactertomatchanyoccurrenceofaletter
followed by “at”. Unfortunately, the replacement string doesn’t match the wildcard
charactervalueofthematchingword.
Usingtheampersand
The sededitorhasasolutionforyou.Theampersandsymbol(&)isusedtorepresentthe
matching pattern in the substitution command. Whatever text matches the pattern
defined,youcanusetheampersandsymboltorecallitinthereplacementpattern.Thislets
youmanipulatewhateverwordmatchesthepatterndefined:
$echo“Thecatsleepsinhishat.”|sed‘s/.at/”&”/g’
The“cat”sleepsinhis“hat”.
$
When the pattern matches the word cat, “cat” appears in the substituted word. When it
matchesthewordhat,“hat”appearsinthesubstitutedword.
Replacingindividualwords
Theampersandsymbolretrievestheentirestringthatmatchesthepatternyouspecifyin
the substitution command. Sometimes, you’ll only want to retrieve a subset of the
string.Youcandothat,too,butit’salittletricky.
The sed editor uses parentheses to define a substring component within the substitution
pattern.Youcanthenreferenceeachsubstringcomponentusingaspecialcharacterinthe
replacementpattern.Thereplacementcharacterconsistsofabackslashandanumber.The
number indicates the substring component’s position. The sed editor assigns the first
componentthecharacter∖1,thesecondcomponentthecharacter∖2,andsoon.
Caution
Whenyouuseparenthesesinthesubstitutioncommand,youmustusetheescape
charactertoidentifythemasgroupingcharactersandnotnormalparentheses.Thisis
thereverseofwhenyouescapeotherspecialcharacters.
Lookatanexampleofusingthisfeatureinasededitorscript:
$echo“TheSystemAdministratormanual”|sed‘
>s/\(System\)Administrator/\1User/’
TheSystemUsermanual
$
This substitution command uses one set of parentheses around the word System
identifyingitasasubstringcomponent.Itthenusesthe ∖1inthereplacementpatternto
recall the first identified component. This isn’t too exciting, but it can really be useful
whenworkingwithwildcardpatterns.
Ifyouneedtoreplaceaphrasewithjustasingleword,that’sasubstringofthephrase,but
thatsubstringjusthappenstobeusingawildcardcharacter;usingsubstringcomponentsis
alifesaver:
$echo“Thatfurrycatispretty”|sed‘s/furry\(.at\)/\1/’
Thatcatispretty
$
$echo“Thatfurryhatispretty”|sed‘s/furry\(.at\)/\1/’
Thathatispretty
$
Inthissituation,youcan’tusetheampersandsymbol,becauseitwouldreplacetheentire
matching pattern. The substring component provides the answer, allowing you to select
justwhichpartofthepatterntouseasthereplacementpattern.
Thisfeaturecanbeespeciallyhelpfulwhenyouneedtoinserttextbetweentwoormore
substringcomponents.Here’sascriptthatusessubstringcomponentstoinsertacommain
longnumbers:
$echo“1234567”|sed‘{
>:start
>s/\(.*[0-9]\)\([0-9]\{3\}\)/\1,\2/
>tstart
>}’
1,234,567
$
Thescriptdividesthematchingpatternintotwocomponents:
.*[0-9]
[0-9]{3}
This pattern looks for two substrings. The first substring is any number of characters,
ending in a digit. The second substring is a series of three digits (see Chapter 20 for
informationabouthowtousebracesinaregularexpression).Ifthispatternisfoundinthe
text,thereplacementtextputsacommabetweenthetwocomponents,eachidentifiedby
itscomponentposition.Thescriptusesthe testcommandtoiteratethroughthenumber
untilallcommashavebeenplaced.
PlacingsedCommandsinScripts
Nowthatyou’veseenthevariouspartsofthesededitor,it’stimetoputthemtogetherand
use them in your shell scripts. This section demonstrates some of the features that you
shouldknowaboutwhenusingthesededitorinyourbashshellscripts.
Usingwrappers
You may have noticed that trying to implement a sed editor script can be cumbersome,
especiallyifthescriptislong.Insteadofhavingtoretypetheentirescripteachtimeyou
want to use it, you can place the sed editor command in a shell script wrapper. The
wrapperactsasago-betweenforthesededitorscriptandthecommandline.
Onceinsidetheshellscript,youcanusenormalshellvariablesandparameterswithyour
sededitorscripts.Here’sanexampleofusingthecommandlineparametervariableasthe
inputtoasedscript:
$catreverse.sh
#!/bin/bash
#Shellwrapperforsededitorscript.
#toreversetextfilelines.
#
sed-n‘{1!G;h;$p}’$1
#
$
The shell script called reverse uses the sed editor script to reverse text lines in a data
stream. It uses the $1 shell parameter to retrieve the first parameter from the command
line,whichshouldbethenameofthefiletoreverse:
$./reverse.shdata2.txt
Thisisthelastline.
Thisistheseconddataline.
Thisisthefirstdataline.
Thisistheheaderline.
$
Now you can easily use the sed editor script on any file, without having to constantly
retypetheentirecommandline.
Redirectingsedoutput
Bydefault,the sededitoroutputstheresultsofthescriptto STDOUT.Youcanemployall
thestandardmethodsofredirectingtheoutputofthesededitorinyourshellscripts.
You can use dollar sign/parenthesis, $(), to redirect the output of your sed editor
commandtoavariableforuselaterinthescript.Thefollowingisanexampleofusingthe
sedscripttoaddcommastotheresultofanumericcomputation:
$catfact.sh
#!/bin/bash
#Addcommastonumberinfactorialanswer
#
factorial=1
counter=1
number=$1
#
while[$counter-le$number]
do
factorial=$[$factorial*$counter]
counter=$[$counter+1]
done
#
result=$(echo$factorial|sed‘{
:start
s/\(.*[0-9]\)\([0-9]\{3\}\)/\1,\2/
tstart
}’)
#
echo“Theresultis$result”
#
$
$./fact.sh20
Theresultis2,432,902,008,176,640,000
$
Afteryouusethenormalfactorialcalculationscript,theresultofthatscriptisusedasthe
input to the sed editor script, which adds commas. This value is then used in the echo
statementtoproducetheresult.
CreatingsedUtilities
Asyou’veseenintheshortexamplespresentedsofarinthischapter,youcandolotsof
cool data-formatting things with the sed editor. This section shows a few handy wellknownsededitorscriptsforperformingcommondata-handlingfunctions.
Spacingwithdoublelines
Tostartthingsoff,lookatasimple sedscripttoinsertablanklinebetweenlinesinatext
file:
$sed‘G’data2.txt
Thisistheheaderline.
Thisisthefirstdataline.
Thisistheseconddataline.
Thisisthelastline.
$
That was pretty simple! The key to this trick is the default value of the hold space.
Remember that the G command simply appends the contents of the hold space to the
currentpatternspacecontents.Whenyoustartthe sededitor,theholdspacecontainsan
emptyline.Byappendingthattoanexistingline,youcreateablanklineaftertheexisting
line.
You may have noticed that this script also adds a blank line to the last line in the data
stream,producingablanklineattheendofthefile.Ifyouwanttogetridofthis,youcan
use the negate symbol and the last line symbol to ensure that the script doesn’t add the
blanklinetothelastlineofthedatastream:
$sed‘$!G’data2.txt
Thisistheheaderline.
Thisisthefirstdataline.
Thisistheseconddataline.
Thisisthelastline.
$
Nowthatlooksalittlebetter.Aslongasthelineisn’tthelastline,theGcommandappends
the contents of the hold space. When the sed editor gets to the last line, it skips the G
command.
Spacingfilesthatmayhaveblanks
Totakedoublespacingonestepfurther,whatifthetextfilealreadyhasafewblanklines,
butyouwanttodoublespaceallthelines?Ifyouusethepreviousscript,you’llgetsome
areasthathavetoomanyblanklines,becauseeachexistingblanklinegetsdoubled:
$catdata6.txt
Thisislineone.
Thisislinetwo.
Thisislinethree.
Thisislinefour.
$
$sed‘$!G’data6.txt
Thisislineone.
Thisislinetwo.
Thisislinethree.
Thisislinefour.
$
Nowyouhavethreeblanklineswheretheoriginalblanklinewaslocated.Thesolutionto
this problem is to first delete any blank lines from the data stream and then use the G
commandtoinsertnewblanklinesafterallthelines.Todeleteexistingblanklines,you
justneedtousethedcommandwithapatternthatmatchesablankline:
/∧$/d
Thispatternusesthestartlinetag(thecaret)andtheendlinetag(thedollarsign).Adding
thispatterntothescriptproducesthedesiredresults:
$sed‘/∧$/d;$!G’data6.txt
Thisislineone.
Thisislinetwo.
Thisislinethree.
Thisislinefour.
$
Perfect!Itworksjustasexpected.
Numberinglinesinafile
Chapter19showedyouhowtousetheequalsigntodisplaythelinenumbersoflinesin
thedatastream:
$sed‘=’data2.txt
1
Thisistheheaderline.
2
Thisisthefirstdataline.
3
Thisistheseconddataline.
4
Thisisthelastline.
$
Thiscanbealittleawkwardtoread,becausethelinenumberisonalineabovetheactual
lineinthedatastream.Abettersolutionistoplacethelinenumberonthesamelineasthe
text.
Nowthatyou’veseenhowtocombinelinesusingtheNcommand,itshouldn’tbetoohard
toutilizethatinformationinthe sededitorscript.Thetricktothisutility,however,isthat
youcan’tcombinethetwocommandsinthesamescript.
Afteryouhavetheoutputfortheequalsigncommand,youcanpipetheoutputtoanother
sededitorscriptthatusesthe Ncommandtocombinethetwolines.Youalsoneedtouse
the substitutioncommandtoreplacethenewlinecharacterwitheitheraspaceoratab
character.Here’swhatthefinalsolutionlookslike:
$sed‘=’data2.txt|sed‘N;s/\n//’
1Thisistheheaderline.
2Thisisthefirstdataline.
3Thisistheseconddataline.
4Thisisthelastline.
$
Nowthatlooksmuchbetter.Thisisagreatlittleutilitytohavearoundwhenworkingon
programswhereyouneedtoseethelinenumbersusedinerrormessages.
Therearebashshellcommandsthatcanalsoaddlinenumbers.However,theyaddsome
additional(andpotentiallyunwantedspacing):
$nldata2.txt
1Thisistheheaderline.
2Thisisthefirstdataline.
3Thisistheseconddataline.
4Thisisthelastline.
$
$cat-ndata2.txt
1Thisistheheaderline.
2Thisisthefirstdataline.
3Thisistheseconddataline.
4Thisisthelastline.
$
Thesededitorscripthandlestheoutputwithoutanyadditionalspacing.
Printinglastlines
Sofar,you’veseenhowtousethepcommandtoprintallthelinesinadatastreamorjust
linesthatmatchaspecificpattern.Whatifyoujustneedtoworkwiththelastfewlinesof
alonglisting,suchasalogfile?
Thedollarsignrepresentsthelastlineofadatastream,soit’seasytodisplayjustthelast
line:
$sed-n‘$p’data2.txt
Thisisthelastline.
$
Nowhowcanyouusethedollarsignsymboltodisplayasetnumberoflinesattheendof
thedatastream?Theansweristocreatearollingwindow.
Arollingwindowisacommonwaytoexamineblocksoftextlinesinthepatternspaceby
combiningthemusingthe Ncommand.The Ncommandappendsthenextlineoftextto
thetextalreadyinthepatternspace.Afteryouhaveablockof10textlinesinthepattern
space,youcanchecktoseeifyou’reattheendofthedatastreamusingthedollarsign.If
you’renotattheend,continueaddingmorelinestothepatternspace,whileremovingthe
originallines(remembertheDcommand,whichdeletesthefirstlineinthepatternspace).
Byloopingthroughthe Nand Dcommands,youaddnewlinestotheblockoflinesinthe
pattern space while removing old lines. The branch command is the perfect fit for the
loop.Toendtheloop,justidentifythelastlineandusetheqcommandtoquit.
Here’swhatthefinalsededitorscriptlookslike:
$catdata7.txt
Thisisline1.
Thisisline2.
Thisisline3.
Thisisline4.
Thisisline5.
Thisisline6.
Thisisline7.
Thisisline8.
Thisisline9.
Thisisline10.
Thisisline11.
Thisisline12.
Thisisline13.
Thisisline14.
Thisisline15.
$
$sed‘{
>:start
>$q;N;11,$D
>bstart
>}’data7.txt
Thisisline6.
Thisisline7.
Thisisline8.
Thisisline9.
Thisisline10.
Thisisline11.
Thisisline12.
Thisisline13.
Thisisline14.
Thisisline15.
$
Thescriptfirstcheckswhetherthelineisthelastlineinthedatastream.Ifitis,the quit
commandstopstheloop.The Ncommandappendsthenextlinetothecurrentlineinthe
patternspace.The 11,$Dcommanddeletesthefirstlineinthepatternspaceifthecurrent
lineisafterline10.Thiscreatestheslidingwindoweffectinthepatternspace.Thus,the
sedprogramscriptdisplaysonlythelast10linesofthedata7.txtfile.
Deletinglines
Another useful utility for the sed editor is to remove unwanted blank lines in a data
stream. It’s easy to remove all the blank lines from a data stream, but it takes a little
ingenuitytoselectivelyremoveblanklines.Thissectionshowsyouacoupleofquick sed
editorscriptsthatyoucanusetohelpremoveunwantedblanklinesfromyourdata.
Deletingconsecutiveblanklines
Itcanbeanuisancewhenextrablanklinescropupindatafiles.Oftenyouhaveadatafile
that contains blank lines, but sometimes a data line is missing and produces too many
blanklines(asyousawinthedouble-spacingexampleearlier).
The easiest way to remove consecutive blank lines is to check the data stream using a
rangeaddress.Chapter19showedyouhowtouserangesinaddresses,includinghowto
incorporate patterns in the address range. The sed editor executes the command for all
linesthatmatchwithinthespecifiedaddressrange.
Thekeytoremovingconsecutiveblanklinesistocreateanaddressrangethatincludesa
non-blank line and a blank line. If the sed editor comes across this range, it shouldn’t
deletetheline.However,forlinesthatdon’tmatchthatrange(twoormoreblanklinesina
row),itshoulddeletethelines.
Here’sthescripttodothis:
/./,/∧$/!d
Therangeis/./to /∧$/.Thestartaddressintherangematchesanylinethatcontainsat
leastonecharacter.Theendaddressintherangematchesablankline.Lineswithinthis
rangearen’tdeleted.
Here’sthescriptinaction:
$catdata8.txt
Thisislineone.
Thisislinetwo.
Thisislinethree.
Thisislinefour.
$
$sed‘/./,/∧$/!d’data8.txt
Thisislineone.
Thisislinetwo.
Thisislinethree.
Thisislinefour.
$
Nomatterhowmanyblanklinesappearbetweenlinesofdatainthefile,theoutputplaces
onlyoneblanklinebetweenthelines.
Deletingleadingblanklines
It is also a nuisance when data files contain multiple blank lines at the start of the file.
Oftenwhenyouaretryingtoimportdatafromatextfileintoadatabase,theblanklines
createnullentries,throwingoffanycalculationsusingthedata.
Removingblanklinesfromthetopofadatastreamisnotadifficulttask.Here’sthescript
thataccomplishesthatfunction:
/./,$!d
Thescriptusesanaddressrangetodeterminewhatlinesaredeleted.Therangestartswith
alinethatcontainsacharacterandcontinuestotheendofthedatastream.Anylinewithin
thisrangeisnotdeletedfromtheoutput.Thismeansthatanylinesbeforethefirstlinethat
containacharacteraredeleted.
Lookatthissimplescriptinaction:
$catdata9.txt
Thisislineone.
Thisislinetwo.
$
$sed‘/./,$!d’data9.txt
Thisislineone.
Thisislinetwo.
$
Thetestfilecontainstwoblanklinesbeforethedatalines.Thescriptsuccessfullyremoves
bothoftheleadingblanklines,whilekeepingtheblanklinewithinthedataintact.
Deletingtrailingblanklines
Unfortunately,deletingtrailingblanklinesisnotassimpleasdeletingleadingblanklines.
Justlikeprintingtheendofadatastream,deletingblanklinesattheendofadatastream
requiresalittleingenuityandlooping.
Beforewestartthediscussion,let’sseewhatthescriptlookslike:
sed‘{
:start
/∧\n*$/{$d;N;bstart}
}’
Thismaylookalittleoddtoyouatfirst.Noticethattherearebraceswithinthenormal
scriptbraces.Thisallowsyoutogroupcommandstogetherwithintheoverallcommand
script. The group of commands applies to the specified address pattern. The address
patternmatchesanylinethatcontainsonlyanewlinecharacter.Whenoneisfound,ifit’s
the last line, the delete command deletes it. If it’s not the last line, the N command
appendsthenextlinetoit,andthebranchcommandloopstothebeginningtostartover.
Here’sthescriptinaction:
$catdata10.txt
Thisisthefirstline.
Thisisthesecondline.
$sed‘{
>:start
>/∧\n*$/{$d;N;bstart}
>}’data10.txt
Thisisthefirstline.
Thisisthesecondline.
$
Thescriptsuccessfullyremovedtheblanklinesfromtheendofthetextfile.
RemovingHTMLtags
Thesedays,it’snotuncommontodownloadtextfromawebsitetosaveoruseasdatain
anapplication.Sometimes,however,whenyoudownloadtextfromthewebsite,youalso
gettheHTMLtagsusedtoformatthedata.Thiscanbeaproblemwhenallyouwantto
seeisthedata.
A standard HTML web page contains several different types of HTML tags, identifying
formattingfeaturesrequiredtoproperlydisplaythepageinformation.Here’sasampleof
whatanHTMLfilelookslike:
$catdata11.txt
<html>
<head>
<titletype=“main”>Thisisthepagetitle</title>
</head>
<body>
<p>
Thisisthe<b>first</b>lineintheWebpage.
Thisshouldprovidesome<i>useful</i>
informationtouseinoursedscript.
</body>
</html>
$
HTML tags are identified by the less-than and greater-than symbols. Most HTML tags
comeinpairs.Onetagstartstheformattingprocess(forexample, <b> for bolding), and
anothertagstopstheformattingprocess(forexample,</b>toturnoffbolding).
RemovingHTMLtagscreatesaproblem,however,ifyou’renotcareful.Atfirstglance,
you’dthinkthatthewaytoremoveHTMLtagswouldbetojustlookforatextstringthat
starts with a less-than symbol (<), ends with a greater-than symbol (>), and has data in
betweenthesymbols:
s/<.*>//g
Unfortunately,thiscommandhassomeunintendedconsequences:
$sed‘s/<.*>//g’data11.txt
ThisisthelineintheWebpage.
Thisshouldprovidesome
informationtouseinoursedscript.
$
Noticethatthetitletextismissing,alongwiththetextthatwasboldedanditalicized.The
sed editor literally interpreted the script to mean any text between the less-than and
greater-thansign,includingotherless-thanandgreater-thansigns!Eachtimethetextwas
enclosedinHTMLtags(suchas<b>first</b>),thesedscriptremovedtheentiretext.
Thesolutiontothisproblemistohavethe sededitorignoreanyembeddedgreater-than
signsbetweentheoriginaltags.Todothat,youcancreateacharacterclassthatnegates
thegreater-thansign.Thischangesthescriptto:
s/<[∧>]*>//g
This script now works properly, displaying the data you need to see from the web page
HTMLcode:
$sed‘s/<[∧>]*>//g’data11.txt
Thisisthepagetitle
ThisisthefirstlineintheWebpage.
Thisshouldprovidesomeuseful
informationtouseinoursedscript.
$
That’salittlebetter.Tocleanthingsupsome,youcanaddadeletecommandtogetridof
thosepeskyblanklines:
$sed‘s/<[∧>]*>//g;/∧$/d’data11.txt
Thisisthepagetitle
ThisisthefirstlineintheWebpage.
Thisshouldprovidesomeuseful
informationtouseinoursedscript.
$
Nowthat’smuchmorecompact;there’sonlythedatayouneedtosee.
Summary
Thesededitorprovidessomeadvancedfeaturesthatallowyoutoworkwithtextpatterns
acrossmultiplelines.Thischaptershowedyouhowtousethe nextcommandtoretrieve
thenextlineinadatastreamandplaceitinthepatternspace.Onceinthepatternspace,
you can perform complex substitution commands to replace phrases that span more
thanonelineoftext.
Themultilinedeletecommandallowsyoutoremovethefirstlinewhenthepatternspace
containstwoormorelines.Thisisaconvenientwaytoiteratethroughmultiplelinesin
thedatastream.Similarly,themultiline printcommandallowsyoutoprintjustthefirst
line when the pattern space contains two or more lines of text. The combination of the
multilinecommandsallowsyoutoiteratethroughthedatastreamandcreateamultiline
substitutionsystem.
Next, we covered the hold space. The hold space allows you to set aside a line of text
whileprocessingmorelinesoftext.Youcanrecallthecontentsoftheholdspaceatany
time and either replace the text in the pattern space or append the contents of the hold
spacetothetextinthepatternspace.Usingtheholdspaceallowsyoutosortthroughdata
streams,reversingtheorderoftextlinesastheyappearinthedata.
Nextwereviewedthevarious sededitorflowcontrolcommands.The branchcommand
provides a way for you to alter the normal flow of sed editor commands in the script,
creating loops or skipping commands under certain conditions. The test command
provides an if-then type of statement for your sed editor command scripts. The test
commandbranchesonlyifapriorsubstitutioncommandsucceedsinreplacingtextina
line.
Thechapterconcludedwithadiscussionofhowtouse sedscriptsinyourshellscripts.A
commontechniqueforlarge sedscriptsistoplacethescriptinashellwrapper.Youcan
use command line parameter variables within the sed script to pass shell command line
values. This creates an easy way to utilize your sed editor scripts directly from the
commandline,orevenfromothershellscripts.
The next chapter digs deeper into the gawk world. The gawk program supports many
featuresofhigher-levelprogramminglanguages.Youcancreatesomeprettyinvolveddata
manipulationandreportingprogramsjustbyusinggawk.Thechapterdescribesthevarious
programming features and demonstrates how to use them to generate your own fancy
reportsfromsimpledata.
Chapter22
Advancedgawk
InThisChapter
1. Reexamininggawk
2. Usingvariablesingawk
3. Usingstructuredcommands
4. Formattingyourprinting
5. Workingwithfunctions
Chapter19introducedthegawkprogramanddemonstratedthebasicsofusingitto
produceformattedreportsfromrawdatafiles.Thischapterdivesmoredeeplyinto
customizinggawktoproducereports.Thegawkprogramisafull-fledged
programminglanguage,providingfeaturesthatallowyoutowriteadvanced
programstomanipulatedata.Ifyouarejumpingintotheshellscriptworldfrom
anotherprogramminglanguage,youshouldfeelrightathomewithgawk.Inthis
chapter,you’llseehowtousethegawkprogramminglanguagetowriteprogramsto
handlejustaboutanydata-formattingtaskyou’llruninto.
UsingVariables
One important feature of any programming language is the ability to store and recall
values using variables. The gawk programming language supports two different types of
variables:
Built-invariables
User-definedvariables
Several built-in variables are available for you to use in gawk. The built-in variables
containinformationusedinhandlingthedatafieldsandrecordsinthedatafile.Youcan
alsocreateyourownvariablesinyour gawkprograms.Thefollowingsectionswalkyou
throughhowtousevariablesinyourgawkprograms.
Built-invariables
Thegawkprogramusesbuilt-invariablestoreferencespecificfeatureswithintheprogram
data. This section describes the built-in variables available for you to use in your gawk
programsanddemonstrateshowtousethem.
Thefieldandrecordseparatorvariables
Chapter 19 demonstrated one type of built-in variable available in gawk: the data field
variables.Thedatafieldvariablesallowyoutoreferenceindividualdatafieldswithina
datarecordusingadollarsignandthenumericalpositionofthedatafieldintherecord.
Thus,toreferencethefirstdatafieldintherecord,youusethe $1variable.Toreference
theseconddatafield,youusethe$2variable,andsoon.
Data fields are delineated by a field separator character. By default, the field separator
characterisawhitespacecharacter,suchasaspaceoratab.Chapter19showedhowto
changethefieldseparatorcharactereitheronthecommandlinebyusingthe-Fcommand
lineparameterorwithinthegawkprogramusingthespecialFSbuilt-invariable.
The FS built-in variable belongs to a group of built-in variables that control how gawk
handlesfieldsandrecordsinbothinputdataandoutputdata.Table22.1liststhebuilt-in
variablescontainedinthisgroup.
Table22.1ThegawkDataFieldandRecordVariables
Variable
Description
Aspace-separatedlistofnumbersdefiningtheexactwidth(inspaces)of
FIELDWIDTHS
eachdatafield
FS
Inputfieldseparatorcharacter
RS
Inputrecordseparatorcharacter
OFS
Outputfieldseparatorcharacter
ORS
Outputrecordseparatorcharacter
The FSand OFS variables define how your gawk program handles data fields in the data
stream.You’vealreadyseenhowtousethe FSvariabletodefinewhatcharacterseparates
datafieldsinarecord.The OFSvariableperformsthesamefunctionbutfortheoutputby
usingtheprintcommand.
Bydefault,gawksetstheOFSvariabletoaspace,sowhenyouusethecommand:
print$1,$2,$3
youseetheoutputas:
field1field2field3
Youcanseethisinthefollowingexample:
$catdata1
data11,data12,data13,data14,data15
data21,data22,data23,data24,data25
data31,data32,data33,data34,data35
$gawk‘BEGIN{FS=”,”}{print$1,$2,$3}’data1
data11data12data13
data21data22data23
data31data32data33
$
TheprintcommandautomaticallyplacesthevalueoftheOFSvariablebetweeneachdata
field in the output. By setting the OFS variable, you can use any string to separate data
fieldsintheoutput:
$gawk‘BEGIN{FS=”,”;OFS=”-“}{print$1,$2,$3}’data1
data11-data12-data13
data21-data22-data23
data31-data32-data33
$gawk‘BEGIN{FS=”,”;OFS=”—”}{print$1,$2,$3}’data1
data11—data12—data13
data21—data22—data23
data31—data32—data33
$gawk‘BEGIN{FS=”,”;OFS=”<—>”}{print$1,$2,$3}’data1
data11<—>data12<—>data13
data21<—>data22<—>data23
data31<—>data32<—>data33
$
The FIELDWIDTHS variable allows you to read records without using a field separator
character.Insomeapplications,insteadofusingafieldseparatorcharacter,dataisplaced
in specific columns within the record. In these instances, you must set the FIELDWIDTHS
variabletomatchthelayoutofthedataintherecords.
After you set the FIELDWIDTHS variable, gawk ignores the FS and calculates data fields
basedontheprovidedfieldwidthsizes.Here’sanexampleusingfieldwidthsinsteadof
fieldseparatorcharacters:
$catdata1b
1005.3247596.37
115-2.349194.00
05810.1298100.1
$gawk‘BEGIN{FIELDWIDTHS=“3525”}{print$1,$2,$3,$4}’data1b
1005.3247596.37
115-2.349194.00
05810.1298100.1
$
The FIELDWIDTHS variable defines four data fields, and gawk parses the data record
accordingly.Thestringofnumbersineachrecordissplitbasedonthedefinedfieldwidth
values.
Caution
It’simportanttorememberthatafteryousettheFIELDWIDTHSvariable,thosevalues
mustremainconstant.Thismethodcan’taccommodatevariable-lengthdatafields.
The RS and ORS variables define how your gawk program handles records in the data
stream.Bydefault,gawksetstheRSandORSvariablestothenewlinecharacter.Thedefault
RS variable value indicates that each new line of text in the input data stream is a new
record.
Sometimes,yourunintosituationswheredatafieldsarespreadacrossmultiplelinesinthe
datastream.Aclassicexampleofthisisdatathatincludesanaddressandphonenumber,
eachonaseparateline:
RileyMullen
123MainStreet
Chicago,IL60601
(312)555-1234
IfyoutrytoreadthisdatausingthedefaultFSandRSvariablevalues,gawkreadseachline
asaseparaterecordandinterpretseachspaceintherecordasafieldseparator.Thisisn’t
whatyouintended.
To solve this problem, you need to set the FS variable to the newline character. This
indicates that each line in the data stream is a separate field and all the data on a line
belongstothedatafield.However,whenyoudothat,youdon’tknowwhereanewrecord
starts.
To solve this problem, set the RS variable to an empty string, and leave a blank line
betweendatarecordsinthedatastream.The gawkprograminterpretseachblanklineasa
recordseparator.
Thefollowingisanexampleofusingthistechnique:
$catdata2
RileyMullen
123MainStreet
Chicago,IL60601
(312)555-1234
FrankWilliams
456OakStreet
Indianapolis,IN46201
(317)555-9876
HaleySnell
4231ElmStreet
Detroit,MI48201
(313)555-4938
$gawk‘BEGIN{FS=”\n”;RS=””}{print$1,$4}’data2
RileyMullen(312)555-1234
FrankWilliams(317)555-9876
HaleySnell(313)555-4938
$
Perfect!The gawk program interpreted each line in the file as a data field and the blank
linesasrecordseparators.
Datavariables
Besides the field and record separator variables, gawk provides some other built-in
variablestohelpyouknowwhat’sgoingonwithyourdataandextractinformationfrom
theshellenvironment.Table22.2showstheotherbuilt-invariablesingawk.
Table22.2MoregawkBuilt-InVariables
Variable
ARGC
ARGIND
ARGV
CONVFMT
ENVIRON
ERRNO
FILENAME
FNR
IGNORECASE
NF
NR
OFMT
RLENGTH
RSTART
Description
Thenumberofcommandlineparameterspresent
TheindexinARGVofthecurrentfilebeingprocessed
Anarrayofcommandlineparameters
Theconversionformatfornumbers(seetheprintfstatement),witha
defaultvalueof%.6g
Anassociativearrayofthecurrentshellenvironmentvariablesandtheir
values
Thesystemerrorifanerroroccurswhenreadingorclosinginputfiles
Thefilenameofthedatafileusedforinputtothegawkprogram
Thecurrentrecordnumberinthedatafile
Ifsettoanon-zerovalue,ignoresthecaseofcharactersinstringsusedin
thegawkcommand
Thetotalnumberofdatafieldsinthedatafile
Thenumberofinputrecordsprocessed
Theoutputformatfordisplayingnumbers,withadefaultof%.6g
Thelengthofthesubstringmatchedinthematchfunction
Thestartindexofthesubstringmatchedinthematchfunction
You should recognize a few of these variables from your shell script programming. The
ARGCandARGVvariablesallowyoutoretrievethenumberofcommandlineparametersand
theirvaluesfromtheshell.Thiscanbealittletricky,however,becausegawkdoesn’tcount
theprogramscriptaspartofthecommandlineparameters:
$gawk‘BEGIN{printARGC,ARGV[1]}’data1
2data1
$
The ARGCvariableindicatesthattwoparametersareonthecommandline.Thisincludes
the gawk command and the data1 parameter (remember that the program script doesn’t
count as a parameter). The ARGV array starts with an index of 0, which represents the
command. The first array value is the first command line parameter after the gawk
command.
Note
Notethatunlikeshellvariables,whenyoureferenceagawkvariableinthescript,you
don’taddadollarsignbeforethevariablename.
TheENVIRONvariablemayseemalittleoddtoyou.Itusesanassociativearraytoretrieve
shell environment variables. An associative array uses text for the array index values
insteadofnumericvalues.
Thetextinthearrayindexistheshellenvironmentvariable.Thevalueofthearrayisthe
valueoftheshellenvironmentvariable.Thefollowingisanexampleofthis:
$gawk‘
>BEGIN{
>printENVIRON[“HOME”]
>printENVIRON[“PATH”]
>}’
/home/rich
/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin
$
The ENVIRON[”HOME”] variable retrieves the HOME environment variable value from the
shell. Likewise, the ENVIRON[”PATH”] variable retrieves the PATH environment variable
value.Youcanusethistechniquetoretrieveanyenvironmentvariablevaluefromtheshell
touseinyourgawkprograms.
The FNR, NF, and NR variables come in handy when you’re trying to keep track of data
fieldsandrecordsinyourgawkprogram.Sometimes,you’reinasituationwhereyoudon’t
knowexactlyhowmanydatafieldsareinarecord.The NFvariableallowsyoutospecify
thelastdatafieldintherecordwithouthavingtoknowitsposition:
$gawk‘BEGIN{FS=”:”;OFS=”:”}{print$1,$NF}’/etc/passwd
rich:/bin/bash
testy:/bin/csh
mark:/bin/bash
dan:/bin/bash
mike:/bin/bash
test:/bin/bash
$
TheNFvariablecontainsthenumericalvalueofthelastdatafieldinthedatafile.Youcan
thenuseitasadatafieldvariablebyplacingadollarsigninfrontofit.
TheFNRandNRvariablesaresimilartoeachother,butslightlydifferent.The FNRvariable
containsthenumberofrecordsprocessedinthecurrentdatafile.TheNRvariablecontains
the total number of records processed. Let’s look at a couple of examples to see this
difference:
$gawk‘BEGIN{FS=”,”}{print$1,“FNR=“FNR}’data1data1
data11FNR=1
data21FNR=2
data31FNR=3
data11FNR=1
data21FNR=2
data31FNR=3
$
Inthisexample,the gawkprogramcommandlinedefinestwoinputfiles.(Itspecifiesthe
sameinputfiletwice.)Thescriptprintsthefirstdatafieldvalueandthecurrentvalueof
the FNR variable. Notice that the FNR value was reset to 1 when the gawk program
processedtheseconddatafile.
Now,let’saddtheNRvariableandseewhatthatproduces:
$gawk‘
>BEGIN{FS=”,”}
>{print$1,“FNR=“FNR,“NR=“NR}
>END{print“Therewere”,NR,“recordsprocessed”}’data1data1
data11FNR=1NR=1
data21FNR=2NR=2
data31FNR=3NR=3
data11FNR=1NR=4
data21FNR=2NR=5
data31FNR=3NR=6
Therewere6recordsprocessed
$
The FNR variable value was reset when gawk processed the second data file, but the NR
variable maintained its count into the second data file. The bottom line is that if you’re
using only one data file for input, the FNR and NR values are the same. If you’re using
multiple data files for input, the FNR value is reset for each data file, and the NR value
keepscountthroughoutallthedatafiles.
Note
Whenusinggawk,noticethatthegawkscriptcanoftenbecomelargerthantherestof
yourshellscript.Intheexamplesinthischapter,forsimplicitywejustrunthegawk
scriptsdirectlyfromthecommandline,usingthemultilinefeatureoftheshell.When
youusegawkinashellscript,youshouldplacedifferentgawkcommandsonseparate
lines.Thismakesitmucheasierforyoutoreadandfollow,ratherthantryingtocram
itallontoonelineintheshellscript.Also,ifyoufindyourselfusingthesamegawk
scriptsindifferentshellscripts,youcansavethegawkscriptinaseparatefileand
referenceitusingthe–fparameter(seeChapter19).
User-definedvariables
Justlikeanyotherself-respectingprogramminglanguage, gawkallowsyoutodefineyour
ownvariablesforusewithintheprogramcode.Agawkuser-definedvariablenamecanbe
any number of letters, digits, and underscores, but it can’t begin with a digit. It is also
importanttorememberthatgawkvariablenamesarecasesensitive.
Assigningvariablesinscripts
Assigning values to variables in gawk programs is similar to doing so in a shell script,
usinganassignmentstatement:
$gawk‘
>BEGIN{
>testing=“Thisisatest”
>printtesting
>}’
Thisisatest
$
Theoutputofthe printstatementisthecurrentvalueofthe testingvariable.Likeshell
scriptvariables,gawkvariablescanholdeithernumericortextvalues:
$gawk‘
>BEGIN{
>testing=“Thisisatest”
>printtesting
>testing=45
>printtesting
>}’
Thisisatest
45
$
In this example, the value of the testing variable is changed from a text value to a
numericvalue.
Assignment statements can also include mathematical algorithms to handle numeric
values:
$gawk‘BEGIN{x=4;x=x*2+3;printx}’
11
$
Asyoucanseefromthisexample,the gawkprogramminglanguageincludesthestandard
mathematicaloperatorsforprocessingnumericalvalues.Thesecanincludetheremainder
symbol(%)andtheexponentiationsymbol(usingeither∧or**).
Assigningvariablesonthecommandline
You can also use the gawk command line to assign values to variables for the gawk
program.Thisallowsyoutosetvaluesoutsideofthenormalcode,changingvaluesonthe
fly.Here’sanexampleofusingacommandlinevariabletodisplayaspecificdatafieldin
thefile:
$catscript1
BEGIN{FS=”,”}
{print$n}
$gawk-fscript1n=2data1
data12
data22
data32
$gawk-fscript1n=3data1
data13
data23
data33
$
Thisfeatureallowsyoutochangethebehaviorofthescriptwithoutnecessitatingthatyou
changetheactualscriptcode.Thefirstexampledisplaystheseconddatafieldinthefile,
whilethesecondexampledisplaysthethirddatafield,justbysettingthevalueofthe n
variableinthecommandline.
There’soneproblemwithusingcommandlineparameterstodefinevariablevalues.When
yousetthevariable,thevalueisn’tavailableintheBEGINsectionofthecode:
$catscript2
BEGIN{print“Thestartingvalueis”,n;FS=”,”}
{print$n}
$gawk-fscript2n=3data1
Thestartingvalueis
data13
data23
data33
$
You can solve this using the -v command line parameter. This allows you to specify
variablesthataresetbeforethe BEGIN section of code. The -vcommandlineparameter
mustbeplacedbeforethescriptcodeinthecommandline:
$gawk-vn=3-fscript2data1
Thestartingvalueis3
data13
data23
data33
$
NowthenvariablecontainsthevaluesetinthecommandlineduringtheBEGINsectionof
code.
WorkingwithArrays
Many programming languages provide arrays for storing multiple values in a single
variable. The gawk programming language provides the array feature using associative
arrays.
Associativearraysaredifferentfromnumericalarraysinthattheindexvaluecanbeany
textstring.Youdon’thavetousesequentialnumberstoidentifydataelementscontained
inthearray.Instead,anassociativearrayconsistsofahodge-podgeofstringsreferencing
values.Eachindexstringmustbeuniqueanduniquelyidentifiesthedataelementthat’s
assigned to it. If you’re familiar with other programming languages, this is the same
conceptashashmapsordictionaries.
The following sections walk you through using associative array variables in your gawk
programs.
Definingarrayvariables
Youcandefineanarrayvariableusingastandardassignmentstatement.Here’stheformat
ofthearrayvariableassignment:
var[index]=element
Inthisexample, varisthevariablename, indexistheassociativearrayindexvalue,and
elementisthedataelementvalue.Herearesomeexamplesofarrayvariablesingawk:
capital[“Illinois”]=“Springfield”
capital[“Indiana”]=“Indianapolis”
capital[“Ohio”]=“Columbus”
When you reference an array variable, you must include the index value to retrieve the
appropriatedataelementvalue:
$gawk‘BEGIN{
>capital[“Illinois”]=“Springfield”
>printcapital[“Illinois”]
>}’
Springfield
$
When you reference the array variable, the data element value appears. This also works
withnumericdataelementvalues:
$gawk‘BEGIN{
>var[1]=34
>var[2]=3
>total=var[1]+var[2]
>printtotal
>}’
37
$
Asyoucanseefromthisexample,youcanusearrayvariablesjustasyouwouldanyother
variableinthegawkprogram.
Iteratingthrougharrayvariables
The problem with associative array variables is that you might not have any way of
knowingwhattheindexvaluesare.Unlikenumericarrays,whichusesequentialnumbers
forindexvalues,anassociativearrayindexcanbeanything.
Ifyouneedtoiteratethroughanassociatearrayin gawk,youcanuseaspecialformatof
theforstatement:
for(varinarray)
{
statements
}
The forstatementloopsthroughthestatements,eachtimeassigningthevariable varthe
next index value from the array associative array. It’s important to remember that the
variableisthevalueoftheindexandnotthedataelementvalue.Youcaneasilyextractthe
dataelementvaluebyusingthevariableasthearrayindex:
$gawk‘BEGIN{
>var[“a”]=1
>var[“g”]=2
>var[“m”]=3
>var[“u”]=4
>for(testinvar)
>{
>print“Index:”,test,”-Value:”,var[test]
>}
>}’
Index:u-Value:4
Index:m-Value:3
Index:a-Value:1
Index:g-Value:2
$
Noticethattheindexvaluesaren’treturnedinanyparticularorder,buttheyeachreference
the appropriate data element value. This is somewhat important to know, because you
can’t count on the returned values being in the same order, just that the index and data
valuesmatch.
Deletingarrayvariables
Removinganarrayindexfromanassociativearrayrequiresaspecialcommand:
deletearray[index]
The delete command removes the associative index value and the associated data
elementvaluefromthearray:
$gawk‘BEGIN{
>var[“a”]=1
>var[“g”]=2
>for(testinvar)
>{
>print“Index:”,test,”-Value:”,var[test]
>}
>deletevar[“g”]
>print“–”
>for(testinvar)
>print“Index:”,test,”-Value:”,var[test]
>}’
Index:a-Value:1
Index:g-Value:2
–
Index:a-Value:1
$
Afteryoudeleteanindexvaluefromtheassociativearray,youcan’tretrieveit.
UsingPatterns
The gawk program supports several types of matching patterns to filter data records, in
muchthesamewayasthe sededitor.Chapter19showedtwospecialpatternsinaction.
The BEGINand ENDkeywordsarespecialpatternsthatexecutestatementsbeforeorafter
the data stream data has been read. Similarly, you can create other patterns to execute
statementswhenmatchingdataappearsinthedatastream.
Thissectiondemonstrateshowtousematchingpatternsinyour gawkscriptstolimitwhat
recordsaprogramscriptappliesto.
Regularexpressions
Chapter 20 showed how to use regular expressions as matching patterns. You can use
either a Basic Regular Expression (BRE) or an Extended Regular Expression (ERE) to
filterwhichlinesinthedatastreamtheprogramscriptappliesto.
Whenusingaregularexpression,theregularexpressionmustappearbeforetheleftbrace
oftheprogramscriptthatitcontrols:
$gawk‘BEGIN{FS=”,”}/11/{print$1}’data1
data11
$
The regular expression /11/ matches records that contain the string 11 anywhere in the
datafields.The gawkprogrammatchesthedefinedregularexpressionagainstallthedata
fieldsintherecord,includingthefieldseparatorcharacter:
$gawk‘BEGIN{FS=”,”}/,d/{print$1}’data1
data11
data21
data31
$
This example matches the comma used as the field separator in the regular expression.
Thisisnotalwaysagoodthing.Itcanleadtoproblemstryingtomatchdataspecificto
onedatafieldthatmayalsoappearinanotherdatafield.Ifyouneedtomatcharegular
expressiontoaspecificdatainstance,youshouldusethematchingoperator.
Thematchingoperator
Thematchingoperatorallowsyoutorestrictaregularexpressiontoaspecificdatafield
intherecords.Thematchingoperatoristhetildesymbol(∼).Youspecifythematching
operator,alongwiththedatafieldvariable,andtheregularexpressiontomatch:
$1~/^data/
The$1variablerepresentsthefirstdatafieldintherecord.Thisexpressionfiltersrecords
wherethefirstdatafieldstartswiththetextdata.Thefollowingisanexampleofusingthe
matchingoperatorinagawkprogramscript:
$gawk‘BEGIN{FS=”,”}$2~/^data2/{print$0}’data1
data21,data22,data23,data24,data25
$
The matching operator compares the second data field with the regular expression /
∧data2/,whichindicatesthestringstartswiththetextdata2.
This is a powerful tool that is commonly used in gawk program scripts to search for
specificdataelementsinadatafile:
$gawk-F:‘$1~/rich/{print$1,$NF}’/etc/passwd
rich/bin/bash
$
Thisexamplesearchesthefirstdatafieldforthetext rich.Whenitfindsthepatternina
record,itprintsthefirstandlastdatafieldvaluesoftherecord.
Youcanalsonegatetheregularexpressionmatchbyusingthe!symbol:
$1!~/expression/
If the regular expression isn’t found in the record, the program script is applied to the
recorddata:
$gawk–F:‘$1!∼/rich/{print$1,$NF}’/etc/passwd
root/bin/bash
daemon/bin/sh
bin/bin/sh
sys/bin/sh
–outputtruncated–
$
Inthisexample,thegawkprogramscriptprintstheuseridandshellforalltheentriesinthe
/etc/passwdfilethatdon’tmatchtheuseridrich!
Mathematicalexpressions
In addition to regular expressions, you can also use mathematical expressions in the
matching pattern. This feature comes in handy when matching numerical values in data
fields. For example, if you want to display all the system users who belong to the root
usersgroup(groupnumber0),youcouldusethisscript:
$gawk-F:‘$4==0{print$1}’/etc/passwd
root
sync
shutdown
halt
operator
$
The script checks for records where the fourth data field contains the value 0. On this
Linuxsystem,fiveuseraccountsbelongtotherootusergroup.
Youcanuseanyofthenormalmathematicalcomparisonexpressions:
x==y:Valuexisequaltoy.
x<=y:Valuexislessthanorequaltoy.
x<y:Valuexislessthany.
x>=y:Valuexisgreaterthanorequaltoy.
x>y:Valuexisgreaterthany.
You can also use expressions with text data, but you must be careful. Unlike regular
expressions,expressionsareanexactmatch.Thedatamustmatchexactlywiththepattern:
$gawk-F,‘$1==“data”{print$1}’data1
$
$gawk-F,‘$1==“data11”{print$1}’data1
data11
$
Thefirsttestdoesn’tmatchanyrecordsbecausethefirstdatafieldvalueisn’tdatainany
oftherecords.Thesecondtestmatchesonerecordwiththevaluedata11.
StructuredCommands
The gawk programming language supports the usual cast of structured programming
commands.Thissectiondescribeseachofthesecommandsanddemonstrateshowtouse
themwithinagawkprogrammingenvironment.
Theifstatement
The gawk programming language supports the standard if-then-else format of the if
statement. You must define a condition for the if statement to evaluate, enclosed in
parentheses. If the condition evaluates to a TRUE condition, the statement immediately
followingtheifstatementisexecuted.IftheconditionevaluatestoaFALSEcondition,the
statementisskipped.Youcanusethisformat:
if(condition)
statement1
Oryoucanplaceitononeline,likethis:
if(condition)statement1
Here’sasimpleexampledemonstratingthisformat:
$catdata4
10
5
13
50
34
$gawk‘{if($1>20)print$1}’data4
50
34
$
Nottoocomplicated.Ifyouneedtoexecutemultiplestatementsinthe ifstatement,you
mustenclosethemwithbraces:
$gawk‘{
>if($1>20)
>{
>x=$1*2
>printx
>}
>}’data4
100
68
$
Becarefulthatyoudon’tconfusetheifstatementbraceswiththebracesusedtostartand
stop the program script. The gawk program can detect missing braces and produces an
errormessageifyoumessup:
$gawk‘{
>if($1>20)
>{
>x=$1*2
>printx
>}’data4
gawk:cmd.line:6:}
gawk:cmd.line:6:^unexpectednewlineorendofstring
$
Thegawkifstatementalsosupportstheelseclause,allowingyoutoexecuteoneormore
statementsiftheifstatementconditionfails.Here’sanexampleofusingtheelseclause:
$gawk‘{
>if($1>20)
>{
>x=$1*2
>printx
>}else
>{
>x=$1/2
>printx
>}}’data4
5
2.5
6.5
100
68
$
Youcanusethe elseclauseonasingleline,butyoumustuseasemicolonafterthe if
statementsection:
if(condition)statement1;elsestatement2
Here’sthesameexampleusingthesinglelineformat:
$gawk‘{if($1>20)print$1*2;elseprint$1/2}’data4
5
2.5
6.5
100
68
$
Thisformatismorecompactbutcanbehardertofollow.
Thewhilestatement
The while statement provides a basic looping feature for gawk programs. Here’s the
formatofthewhilestatement:
while(condition)
{
statements
}
Thewhileloopallowsyoutoiterateoverasetofdata,checkingaconditionthatstopsthe
iteration.Thisisusefulifyouhavemultipledatavaluesineachrecordthatyoumustuse
incalculations:
$catdata5
130120135
160113140
145170215
$gawk‘{
>total=0
>i=1
>while(i<4)
>{
>total+=$i
>i++
>}
>avg=total/3
>print“Average:”,avg
>}’data5
Average:128.333
Average:137.667
Average:176.667
$
Thewhilestatementiteratesthroughthedatafieldsintherecord,addingeachvaluetothe
totalvariableandincrementingthecountervariable,i.Whenthecountervalueisequalto
4,the whileconditionbecomes FALSE,andtheloopterminates,droppingthroughtothe
nextstatementinthescript.Thatstatementcalculatestheaverageandprintstheaverage.
Thisprocessisrepeatedforeachrecordinthedatafile.
The gawk programming language supports using the break and continue statements in
whileloops,allowingyoutojumpoutofthemiddleoftheloop:
$gawk‘{
>total=0
>i=1
>while(i<4)
>{
>total+=$i
>if(i==2)
>break
>i++
>}
>avg=total/2
>print“Theaverageofthefirsttwodataelementsis:”,avg
>}’data5
Theaverageofthefirsttwodataelementsis:125
Theaverageofthefirsttwodataelementsis:136.5
Theaverageofthefirsttwodataelementsis:157.5
$
Thebreakstatementisusedtobreakoutofthewhileloopifthevalueoftheivariableis
2.
Thedo-whilestatement
The do-while statement is similar to the while statement but performs the statements
beforecheckingtheconditionstatement.Here’stheformatforthedo-whilestatement:
do
{
statements
}while(condition)
This format guarantees that the statements are executed at least one time before the
conditionisevaluated.Thiscomesinhandywhenyouneedtoperformstatementsbefore
evaluatingthecondition:
$gawk‘{
>total=0
>i=1
>do
>{
>total+=$i
>i++
>}while(total<150)
>printtotal}’data5
250
160
315
$
Thescriptreadsthedatafieldsfromeachrecordandtotalsthemuntilthecumulativevalue
reaches150.Ifthefirstdatafieldisover150(asseeninthesecondrecord),thescriptis
guaranteedtoreadatleastthefirstdatafieldbeforeevaluatingthecondition.
Theforstatement
The for statement is a common method used in many programming languages for
looping.ThegawkprogramminglanguagesupportstheC-styleofforloops:
for(variableassignment;condition;iterationprocess)
Thishelpssimplifytheloopbycombiningseveralfunctionsinonestatement:
$gawk‘{
>total=0
>for(i=1;i<4;i++)
>{
>total+=$i
>}
>avg=total/3
>print“Average:”,avg
>}’data5
Average:128.333
Average:137.667
Average:176.667
$
By defining the iteration counter in the for loop, you don’t have to worry about
incrementingityourselfasyoudidwhenusingthewhilestatement.
FormattedPrinting
Youmayhavenoticedthattheprintstatementdoesn’texactlygiveyoumuchcontrolover
how gawk displays your data. About all you can do is control the output field separator
character (OFS). If you’re creating detailed reports, often you need to place data in a
specificformatandlocation.
Thesolutionistousetheformattedprintingcommand,called printf.Ifyou’refamiliar
withCprogramming,the printfcommandin gawkperformsthesameway,allowingyou
tospecifydetailedinstructionsonhowtodisplaydata.
Here’stheformatoftheprintfcommand:
printf“formatstring“,var1,var2…
Theformatstringisthekeytotheformattedoutput.Itspecifiesexactlyhowtheformatted
outputshouldappear,usingbothtextelementsandformatspecifiers.Aformatspecifieris
aspecialcodethatindicateswhattypeofvariableisdisplayedandhowtodisplayit.The
gawk program uses each format specifier as a placeholder for each variable listed in the
command.Thefirstformatspecifiermatchesthefirstvariablelisted,thesecondmatches
thesecondvariable,andsoon.
Theformatspecifiersusethefollowingformat:
%[modifier]control-letter
Inthisexample, control-letterisaone-charactercodethatindicateswhattypeofdata
valuewillbedisplayed,andmodifierdefinesanoptionalformattingfeature.
Table22.3liststhecontrollettersthatcanbeusedintheformatspecifier.
Table22.3FormatSpecifierControlLetters
ControlLetter
c
d
i
e
f
g
o
s
x
X
Description
DisplaysanumberasanASCIIcharacter
Displaysanintegervalue
Displaysanintegervalue(sameasd)
Displaysanumberinscientificnotation
Displaysafloating-pointvalue
Displayseitherscientificnotationorfloatingpoint,whicheverisshorter
Displaysanoctalvalue
Displaysatextstring
Displaysahexadecimalvalue
Displaysahexadecimalvalue,butusingcapitallettersforAthroughF
Thus,ifyouneedtodisplayastringvariable,youusetheformatspecifier %s.Ifyouneed
todisplayanintegervariable,youuseeither %dor %i(%distheC-stylefordecimals).If
youwanttodisplayalargevalueusingscientificnotation,youusethe%eformatspecifier:
$gawk‘BEGIN{
>x=10*100
>printf“Theansweris:%e\n”,x
>}’
Theansweris:1.000000e+03
$
Inadditiontothecontrolletters,youcanusethreemodifiersforevenmorecontrolover
youroutput:
width:Thisisanumericvaluethatspecifiestheminimumwidthoftheoutputfield.
Iftheoutputisshorter,printfpadsthespacewithspaces,usingrightjustification
forthetext.Iftheoutputislongerthanthespecifiedwidth,itoverridesthewidth
value.
prec:Thisisanumericvaluethatspecifiesthenumberofdigitstotherightofthe
decimalplaceinfloating-pointnumbers,orthemaximumnumberofcharacters
displayedinatextstring.
-(minussign):Theminussignindicatesthatleftjustificationshouldbeusedinstead
ofrightjustificationwhenplacingdataintheformattedspace.
When using the printf statement, you have complete control over how your output
appears.Forexample,inthe“Built-invariables”section,weusedthe printcommandto
displaydatafieldsfromourrecords:
$gawk‘BEGIN{FS=”\n”;RS=””}{print$1,$4}’data2
RileyMullen(312)555-1234
FrankWilliams(317)555-9876
HaleySnell(313)555-4938
$
Youcanusethe printfcommandtohelpformattheoutputsoitlooksbetter.First,let’s
justconverttheprintcommandtoaprintfcommandandseewhatthatdoes:
$gawk‘BEGIN{FS=”\n”;RS=””}{printf“%s%s\n”,$1,$4}’data2
RileyMullen(312)555-1234
FrankWilliams(317)555-9876
HaleySnell(313)555-4938
$
Thatproducesthesameoutputasthe printcommand.The printfcommandusesthe %s
formatspecifierasaplaceholderforthetwostringvalues.
Notice that you have to manually add the newline character at the end of the printf
command to force a new line. Without it, the printf command uses the same line on
subsequentprints.
This is useful if you need to print multiple things on the same line, but using separate
printfcommands:
$gawk‘BEGIN{FS=”,”}{printf“%s“,$1}END{printf“\n”}’data1
data11data21data31
$
Both printf outputs appear on the same line. To be able to terminate the line, the END
sectionprintsasinglenewlinecharacter.
Next,let’suseamodifiertoformatthefirststringvalue:
$gawk‘BEGIN{FS=”\n”;RS=””}{printf“%16s%s\n”,$1,$4}’data2
RileyMullen(312)555-1234
FrankWilliams(317)555-9876
HaleySnell(313)555-4938
$
Byaddingthe 16modifiervalue,weforcetheoutputforthefirststringtouse16spaces.
By default, the printf command uses right justification to place the data in the format
space.Tomakeitleftjustified,justaddaminussigntothemodifier:
$gawk‘BEGIN{FS=”\n”;RS=””}{printf“%-16s%s\n”,$1,$4}’data2
RileyMullen(312)555-1234
FrankWilliams(317)555-9876
HaleySnell(313)555-4938
$
Nowthatlooksprettyprofessional!
The printfcommandalsocomesinhandywhendealingwithfloating-pointvalues.By
specifyingaformatforthevariable,youcanmaketheoutputlookmoreuniform:
$gawk‘{
>total=0
>for(i=1;i<4;i++)
>{
>total+=$i
>}
>avg=total/3
>printf“Average:%5.1f\n”,avg
>}’data5
Average:128.3
Average:137.7
Average:176.7
$
By using the %5.1f format specifier, you can force the printf command to round the
floating-pointvaluestoasingledecimalplace.
Built-InFunctions
The gawk programming language provides quite a few built-in functions that perform
commonmathematical,string,andeventimefunctions.Youcanutilizethesefunctionsin
your gawk programs to help cut down on the coding requirements in your scripts. This
sectionwalksyouthroughthedifferentbuilt-infunctionsavailableingawk.
Mathematicalfunctions
Ifyou’vedoneprogramminginanytypeoflanguage,you’reprobablyfamiliarwithusing
built-in functions in your code to perform common mathematical functions. The gawk
programming language doesn’t disappoint those looking for advanced mathematical
features.
Table22.4showsthemathematicalbuilt-infunctionsavailableingawk.
Table22.4ThegawkMathematicalFunctions
Function
Description
atan2(x,y) Thearctangentofx/y,withxandyspecifiedinradians
cos(x)
Thecosineofx,withxspecifiedinradians
exp(x)
Theexponentialofx
int(x)
Theintegerpartofx,truncatedtoward0
log(x)
Thenaturallogarithmofx
rand()
Arandomfloatingpointvaluelargerthan0andlessthan1
sin(x)
Thesineofx,withxspecifiedinradians
sqrt(x)
Thesquarerootofx
srand(x)
Specifiesaseedvalueforcalculatingrandomnumbers
Althoughitdoesnothaveanextensivelistofmathematicalfunctions, gawkdoesprovide
some of the basic elements you need for standard mathematical processing. The int()
functionproducestheintegerportionofavalue,butitdoesn’troundthevalue.Itbehaves
muchlikeafloorfunctionfoundinotherprogramminglanguages.Itproducesthenearest
integertoavaluebetweenthevalueand0.
Thismeansthattheint()functionofthevalue5.6returns5,whilethe int()functionof
thevalue-5.6returns-5.
The rand()functionisgreatforcreatingrandomnumbers,butyouneedtouseatrickto
getmeaningfulvalues.The rand()functionreturnsarandomnumber,butonlybetween
thevalues0and1(notincluding0or1).Togetalargernumber,youneedtoscalethe
returnedvalue.
Acommonmethodforproducinglargerintegerrandomnumbersistocreateanalgorithm
thatusestherand()function,alongwiththeint()function:
x=int(10*rand())
Thisreturnsarandomintegervaluebetween(andincluding)0and9.Justsubstitutethe10
intheequationwiththeupperlimitvalueforyourapplication,andyou’rereadytogo.
Be careful when using some of the mathematical functions, because the gawk
programminglanguagedoeshavealimitedrangeofnumericvaluesitcanworkwith.If
yougooverthatrange,yougetanerrormessage:
$gawk‘BEGIN{x=exp(100);printx}’
26881171418161356094253400435962903554686976
$gawk‘BEGIN{x=exp(1000);printx}’
gawk:warning:expargument1000isoutofrange
inf
$
Thefirstexamplecalculatesthenaturalexponentialfunctionof100,whichisaverylarge
numberbutwithintherangeofthesystem.Thesecondexampleattemptstocalculatethe
natural exponential function of 1,000, which goes over the numerical range limit of the
systemandproducesanerrormessage.
Besides the standard mathematical functions, gawk also provides a few functions for
bitwisemanipulatingofdata:
and(v1,v2):PerformsabitwiseANDofvaluesv1andv2
compl(val):Performsthebitwisecomplementofval
lshift(val,count):Shiftsthevaluevalcountnumberofbitsleft
or(v1,v2):PerformsabitwiseORofvaluesv1andv2
rshift(val,count):Shiftsthevaluevalcountnumberofbitsright
xor(v1,v2):PerformsabitwiseXORofvaluesv1andv2
Thebitmanipulationfunctionsareusefulwhenworkingwithbinaryvaluesinyourdata.
Stringfunctions
The gawk programming language also provides several functions you can use to
manipulatestringvalues,showninTable22.5.
Table22.5ThegawkStringFunctions
Function
Description
Thisfunctionsortsanarraysbasedonthedataelementvalues.The
indexvaluesarereplacedwithsequentialnumbersindicatingthenew
asort(s[,d])
sortorder.Alternatively,thenewsortedarrayisstoredinarraydif
specified.
Thisfunctionsortsanarraysbasedontheindexvalues.Theresulting
arraycontainstheindexvaluesasthedataelementvalues,with
asorti(s[,d])
sequentialnumberindexesindicatingthesortorder.Alternatively,the
newsortedarrayisstoredinarraydifspecified.
Thisfunctionsearcheseitherthevariable$0,orthetargetstringtif
supplied,formatchesoftheregularexpressionr.Ifhisastring
beginningwitheithergorG,itreplacesthematchingtextwiths.Ifh
isanumber,itrepresentswhichoccurrenceofrtoreplace.
Thisfunctionsearcheseitherthevariable$0,orthetargetstringtif
gsub(r,s[,t])
supplied,formatchesoftheregularexpressionr.Iffound,it
substitutesthestringsglobally.
Thisfunctionreturnstheindexofthestringtinstrings,or0ifnot
index(s,t)
found.
Thisfunctionreturnsthelengthofstrings,orifnotspecified,the
length([s])
lengthof$0.
Thisfunctionreturnstheindexofthestringswheretheregular
match(s,r
expressionroccurs.Ifarrayaisspecified,itcontainstheportionofs
[,a])
thatmatchestheregularexpression.
ThisfunctionsplitssintoarrayausingeithertheFScharacter,orthe
split(s,a
[,r])
regularexpressionrifsupplied.Itreturnsthenumberoffields.
sprintf(format, Thisfunctionreturnsastringsimilartotheoutputofprintfusingthe
variables)
formatandvariablessupplied.
Thisfunctionsearcheseitherthevariable$0,orthetargetstringt,for
sub(r,s[,t]) matchesoftheregularexpressionr.Iffound,itsubstitutesthestrings
forthefirstoccurrence.
Thisfunctionreturnsthenthcharactersubstringofs,startingatindex
substr(s,i
[,n])
i.Ifnisnotsupplied,therestofsisused.
tolower(s)
Thisfunctionconvertsallcharactersinstolowercase.
toupper(s)
Thisfunctionconvertsallcharactersinstouppercase.
gensub(r,s,h
[,t])
Somestringfunctionsarerelativelyself-explanatory:
$gawk‘BEGIN{x=“testing”;printtoupper(x);printlength(x)}’
TESTING
7
$
However, some string functions can get pretty complicated. The asort and asorti
functionsarenew gawkfunctionsthatallowyoutosortanarrayvariablebasedoneither
thedataelementvalues(asort)ortheindexvalues(asorti).Here’sanexampleofusing
asort:
$gawk‘BEGIN{
>var[“a”]=1
>var[“g”]=2
>var[“m”]=3
>var[“u”]=4
>asort(var,test)
>for(iintest)
>print“Index:”,i,”-value:”,test[i]
>}’
Index:4-value:4
Index:1-value:1
Index:2-value:2
Index:3-value:3
$
Thenewarray,test,containsthenewlysorteddataelementsoftheoriginalarray,butthe
indexvaluesarenowchangedtonumericalvalues,indicatingthepropersortorder.
Thesplitfunctionisagreatwaytopushdatafieldsintoanarrayforfurtherprocessing:
$gawk‘BEGIN{FS=”,”}{
>split($0,var)
>printvar[1],var[5]
>}’data1
data11data15
data21data25
data31data35
$
The new array uses sequential numbers for the array index, starting with index value 1
containingthefirstdatafield.
Timefunctions
The gawk programming language contains a few functions to help you deal with time
values,showninTable22.6.
Table22.6ThegawkTimeFunctions
Function
Description
ConvertsadatespecifiedintheformatYYYYMMDDHHMMSS
mktime(datespec)
[DST]intoatimestampvalue
Formatseitherthecurrenttimeofdaytimestamp,ortimestampif
strftime(format
provided,intoaformatteddayanddate,usingthedate()shell
[,timestamp])
functionformat
systime()
Returnsthetimestampforthecurrenttimeofday
Thetimefunctionsareoftenusedwhenworkingwithlogfilesthatcontaindatesthatyou
need to compare. By converting the text representation of a date to the epoch time (the
numberofsecondssincemidnight,January1,1970),youcaneasilycomparedates.
Thefollowingisanexampleofusingthetimefunctionsinagawkprogram:
$gawk‘BEGIN{
>date=systime()
>day=strftime(“%A,%B%d,%Y”,date)
>printday
>}’
Friday,December26,2014
$
Thisexampleusesthe systimefunctiontoretrievethecurrentepochtimestampfromthe
system and then uses the strftime function to convert it into a human-readable format
usingthedateshellcommand’sdateformatcharacters.
User-DefinedFunctions
You’re not limited to just using the built-in functions available in gawk. You can create
yourownfunctionsforusein gawkprograms.Thissectionshowsyouhowtodefineand
useyourownfunctionsingawkprograms.
Definingafunction
Todefineyouownfunction,youmustusethefunctionkeyword:
functionname([variables])
{
statements
}
The function name must uniquely identify your function. You can pass one or more
variablesintothefunctionfromthecallinggawkprogram:
functionprintthird()
{
print$3
}
Thisfunctionprintsthethirddatafieldintherecord.
Thefunctioncanalsoreturnavalueusingthereturnstatement:
returnvalue
Thevaluecanbeavariable,oranequationthatevaluatestoavalue:
functionmyrand(limit)
{
returnint(limit*rand())
}
Youcanassignthevaluereturnedfromthefunctiontoavariableinthegawkprogram:
x=myrand(100)
Thevariablecontainsthevaluereturnedfromthefunction.
Usingyourfunctions
Whenyoudefineafunction,itmustappearbyitselfbeforeyoudefineanyprogramming
sections(includingtheBEGINsection).Thismaylookalittleoddatfirst,butithelpskeep
thefunctioncodeseparatefromtherestofthegawkprogram:
$gawk‘
>functionmyprint()
>{
>printf“%-16s-%s\n”,$1,$4
>}
>BEGIN{FS=”\n”;RS=””}
>{
>myprint()
>}’data2
RileyMullen-(312)555-1234
FrankWilliams-(317)555-9876
HaleySnell-(313)555-4938
$
Thefunctiondefinesthemyprint()function,whichformatsthefirstandfourthdatafields
in the record for printing. The gawk program then uses the function to display the data
fromthedatafile.
Afteryoudefineafunction,youcanuseitasoftenasnecessaryintheprogramsectionof
thecode.Thissaveslotsofworkwhenusinglongalgorithms.
Creatingafunctionlibrary
Obviously, having to rewrite your gawk functions every time you need them is not a
pleasant experience. However, gawk provides a way for you to combine your functions
intoasinglelibraryfilethatyoucanuseinallyourgawkprogramming.
First,youneedtocreateafilethatcontainsallyourgawkfunctions:
$catfunclib
functionmyprint()
{
printf“%-16s-%s\n”,$1,$4
}
functionmyrand(limit)
{
returnint(limit*rand())
}
functionprintthird()
{
print$3
}
$
The funclibfilecontainsthreefunctiondefinitions.Tousethem,youneedtousethe -f
command line parameter. Unfortunately, you can’t combine the -f command line
parameterwithaninline gawkscript,butyoucanusemultiple -fparametersonthesame
commandline.
Thus,touseyourlibrary,justcreateafilethatcontainsyour gawkprogram,andspecify
boththelibraryfileandyourprogramfileonthecommandline:
$catscript4
BEGIN{FS=”\n”;RS=””}
{
myprint()
}
$gawk-ffunclib-fscript4data2
RileyMullen-(312)555-1234
FrankWilliams-(317)555-9876
HaleySnell-(313)555-4938
$
Nowyoujustneedtoaddthefunclibfiletoyourgawkcommandlinewheneveryouneed
touseafunctiondefinedinthelibrary.
WorkingthroughaPracticalExample
Theadvancedgawkfeaturescomeinhandyifyouhavetohandledatavaluesinadatafile,
suchastabulatingsalesfiguresorcalculatingbowlingscores.Whenyouworkwithdata
files, the key is to first group related data records together and then perform any
calculationsrequiredontherelateddata.
For example, let’s work with a data file that contains the bowling scores from a game
betweentwoteams,eachwithtwoplayers:
$catscores.txt
RichBlum,team1,100,115,95
BarbaraBlum,team1,110,115,100
ChristineBresnahan,team2,120,115,118
TimBresnahan,team2,125,112,116
$
Each player has scores from three separate games in the data file, and each player is
identifiedbyateamnameinthesecondcolumn.Here’stheshellscripttosortthedatafor
eachteamandcalculatethetotalsandaverages:
$catbowling.sh
#!/bin/bash
forteamin$(gawk–F,‘{print$2}’scores.txt|uniq)
do
gawk–vteam=$team‘BEGIN{FS=”,”;total=0}
{
if($2==team)
{
total+=$3+$4+$5;
}
}
END{
avg=total/6;
print“Totalfor”,team,“is”,total,“,theaverageis”,avg
}’scores.txt
done
$
Thefirstgawkstatementinsidetheforloopfiltersouttheteamnamesinthedatafileand
thenusestheuniqfunctiontoreturnonevalueforeachseparateteamname.Theforloop
theniteratesforeachseparateteamname.
The gawk statement inside the for loop is what’s doing the calculations. For each data
record,itfirstdeterminesiftheteamnamematchestheloopteam.That’sdonebyusing
the–voptionin gawk,whichallowsustopassashellvariableinsidethe gawkprogram.If
the team name matches, the code keeps a running sum of the three scores in the data
record, adding each data record’s values, as long as that data record matches the team
name.
Attheendofeachloopiteration,the gawk code displays the score totals, as well as the
averageofthescores.Theoutputshouldlooklikethis:
$./bowling.sh
Totalforteam1is635,theaverageis105.833
Totalforteam2is706,theaverageis117.667
$
Now you have a handy shell script to calculate the results of all your bowling
tournaments;youjustneedtoplugthedatafromeachplayerintothedatatextfileandrun
thescript!
Summary
This chapter walked you through the more advanced features of the gawk programming
language.Everyprogramminglanguagerequiresusingvariables,and gawkisnodifferent.
The gawk programming language includes some built-in variables that you can use to
reference specific data field values and retrieve information about the number of data
fieldsandrecordsprocessedinthedatafile.Youcanalsocreateyourownvariablesfor
useinyourscripts.
Thegawkprogramminglanguagealsoprovidesmanyofthestandardstructuredcommands
you expect from a programming language. You can easily create fancy programs using
if-thenlogicand while, do-while,and forloops.Eachofthesecommandsallowsyou
toaltertheflowofyour gawkprogramscripttoiteratethroughdatafieldvaluestocreate
detaileddatareports.
Theprintfcommandisagreattooltohaveifyouneedtocustomizeyourreportoutput.
Itallowsyoutospecifytheexactformatfordisplayingdatafromthegawkprogramscript.
You can easily create formatted reports, placing data elements in exactly the correct
position.
Finally, this chapter discussed the many built-in functions available in the gawk
programming language and showed you how to create your own functions. The gawk
program contains many useful functions for handling mathematical features, such as
standard square roots and logarithms, as well as trigonometric functions. There are also
several string-related functions that make extracting substrings from larger strings a
breeze.
Youaren’tlimitedtothebuilt-infunctionsinthe gawkprogram.Ifyou’reworkingonan
applicationthatuseslotsofspecializedalgorithms,youcancreateyourownfunctionsto
processthealgorithmsandusethosefunctionsinyourowncode.Youcanalsosetupa
libraryfilecontainingallthefunctionsyouuseinyour gawkprograms,savingyoutime
andeffortinallyourcoding.
Thenextchapterswitchesgearsalittle.Itexaminesafewothershellenvironmentsyou
mayrunintoinyourLinuxshell-scriptingendeavors.Althoughthebashshellisthemost
commonshellusedinLinux,it’snottheonlyshell.Ithelpstoknowalittleaboutsomeof
theothershellsavailableandhowtheydifferfromthebashshell.
Chapter23
WorkingwithAlternativeShells
InThisChapter
1. Understandingthedashshell
2. Programminginthedashshell
3. Introducingthezshshell
4. Writingscriptsforzsh
AlthoughthebashshellisthemostwidelyusedshellinLinuxdistributions,itisn’t
theonlyone.Nowthatyou’veseenthestandardLinuxbashshellandwhatyoucan
dowithit,it’stimetoexamineafewothershellsavailableintheLinuxworld.This
chapterdescribestwoothershellsthatyoumayrunintoinyourLinuxjourneyand
howtheydifferfromthebashshell.
WhatIsthedashShell?
TheDebiandashshellhashadaninterestingpast.It’sadirectdescendantoftheashshell,
a simple copy of the original Bourne shell available on Unix systems (see Chapter 1).
KennethAlmquistcreatedasmall-scaleversionoftheBourneshellforUnixsystemsand
calledittheAlmquistshell,whichwasthenshortenedtoash.Thisoriginalversionofthe
ash shell was extremely small and fast but without many advanced features, such as
commandlineeditingorhistoryfeatures,makingitdifficulttouseasaninteractiveshell.
The NetBSD Unix operating system adopted the ash shell and still uses it today as the
default shell. The NetBSD developers customized the ash shell by adding several new
features, making it closer to the Bourne shell. The new features include command line
editingusingbothemacsandvieditorcommands,aswellasahistorycommandtorecall
previouslyenteredcommands.ThisversionoftheashshellisalsousedbytheFreeBSD
operatingsystemasthedefaultloginshell.
TheDebianLinuxdistributioncreateditsownversionoftheashshell(calledDebianash,
ordash)forinclusioninitsversionofLinux.Forthemostpart,dashcopiesthefeaturesof
the NetBSD version of the ash shell, providing the advanced command line editing
capabilities.
However,toaddtotheshellconfusion,thedashshellisactuallynotthedefaultshellin
many Debian-based Linux distributions. Because of the popularity of the bash shell in
Linux,mostDebian-basedLinuxdistributionsusethebashshellasthenormalloginshell
and use the dash shell only as a quick-start shell for the installation script to install the
distributionfiles.
The exception is the popular Ubuntu distribution. This often confuses shell script
programmersandcausesagreatnumberofproblemswithrunningshellscriptsinaLinux
environment.TheUbuntuLinuxdistributionusesthebashshellasthedefaultinteractive
shell,butusesthedashshellasthedefault /bin/shshell.This“feature”reallyconfuses
shellscriptprogrammers.
AsyousawinChapter11,everyshellscriptmuststartwithalinethatdeclarestheshell
usedforthescript.Inourbashshellscripts,we’vebeenusingthis:
#!/bin/bash
Thistellstheshelltousetheshellprogramlocatedat /bin/bashtoexecutethescript.In
the Unix world, the default shell was always /bin/sh. Many shell script programmers
familiarwiththeUnixenvironmentcopythisintotheirLinuxshellscripts:
#!/bin/sh
On most Linux distributions, the /bin/sh file is a symbolic link (see Chapter 3) to the
/bin/bash shell program. This allows you to easily port shell scripts designed for the
UnixBourneshelltotheLinuxenvironmentwithouthavingtomodifythem.
Unfortunately,theUbuntuLinuxdistributionlinksthe/bin/shfiletothe/bin/dashshell
program.Becausethedashshellcontainsonlyasubsetofthecommandsavailableinthe
originalBourneshell,thiscan—andoftendoes—causesomeshellscriptstonotwork
properly.
Thenextsectionwalksyouthroughthebasicsofthedashshellandhowitdiffersfromthe
bash shell. This is especially important to know if you write bash shell scripts that may
needtoberuninanUbuntuenvironment.
ThedashShellFeatures
AlthoughboththebashshellandthedashshellaremodeledaftertheBourneshell,they
havesomedifferences.ThissectionwalksyouthroughthefeaturesfoundintheDebian
dash shell to acquaint you with how the dash shell works before we dive into the shell
scriptingfeatures.
Thedashcommandlineparameters
Thedashshellusescommandlineparameterstocontrolitsbehavior.Table23.1liststhe
commandlineparametersanddescribeswhateachonedoes.
Table23.1ThedashCommandLineParameters
Parameter
-a
-c
-e
-f
-n
-u
-v
-x
-I
-i
-m
-s
-E
-V
Description
Exportsallvariablesassignedtotheshell
Readscommandsfromaspecifiedcommandstring
Ifnotinteractive,exitsimmediatelyifanyuntestedcommandfails
Displayspathnamewildcardcharacters
Ifnotinteractive,readscommandsbutdoesn’texecutethem
WritesanerrormessagetoSTDERRwhenattemptingtoexpandavariablethat
isnotset
WritesinputtoSTDERRasitisread
WriteseachcommandtoSTDERRasitisexecuted
IgnoresEOFcharactersfromtheinputwhenininteractivemode
Forcestheshelltooperateininteractivemode
Turnsonjobcontrol(enabledbydefaultininteractivemode)
ReadscommandsfromSTDIN(thedefaultbehaviorifnofileargumentsare
present)
Enablestheemacscommandlineeditor
Enablesthevicommandlineeditor
Debian added a few additional command line parameters to the original ash shell
commandlineparameterlist.The -Eand -Vcommandlineparametersenablethespecial
commandlineeditingfeaturesofthedashshell.
The-Ecommandlineparameterallowsyoutousetheemacseditorcommandsforediting
command line text (see Chapter 10). You can use all the emacs commands for
manipulatingtextonasinglelineusingtheCtrlandMetakeycombinations.
The -V command line parameter allows you to use the vi editor commands for editing
command line text (again, see Chapter 10). This feature allows you to switch between
normalmodeandvieditormodeonthecommandlinebyusingtheEsckey.Whenyou’re
invieditormode,youcanuseallthestandardvieditorcommands(suchas xtodeletea
character,anditoinserttext).Afteryoufinisheditingthecommandline,youmustpress
theEsckeyagaintoexitvieditormode.
Thedashenvironmentvariables
Thedashshellusesquiteafewdefaultenvironmentvariablesusestotrackinformation,
and you can create your own environment variables as well. This section describes the
environmentvariablesandhowdashhandlesthem.
Defaultenvironmentvariables
Thedashenvironmentvariablesareverysimilartotheenvironmentvariablesusedinbash
(seeChapter6).Thisisnotbyaccident.Rememberthatboththedashandbashshellsare
extensions of the Bourne shell, so they both incorporate many of its features. However,
becauseofitsgoalofsimplicity,thedashshellcontainssignificantlyfewerenvironment
variablesthanthebashshell.Youneedtotakethisintoconsiderationwhencreatingshell
scriptsinadashshellenvironment.
Thedashshellusesthesetcommandtodisplayenvironmentvariables:
$set
COLORTERM=”
DESKTOP_SESSION=‘default’
DISPLAY=’:0.0’
DM_CONTROL=’/var/run/xdmctl’
GS_LIB=’/home/atest/.fonts’
HOME=’/home/atest’
IFS=’
’
KDEROOTHOME=’/root/.kde’
KDE_FULL_SESSION=‘true’
KDE_MULTIHEAD=‘false’
KONSOLE_DCOP=‘DCOPRef(konsole-5293,konsole)’
KONSOLE_DCOP_SESSION=‘DCOPRef(konsole-5293,session-1)’
LANG=‘en_US’
LANGUAGE=‘en’
LC_ALL=‘en_US’
LOGNAME=‘atest’
OPTIND=‘1’
PATH=’/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin’
PPID=‘5293’
PS1=’$‘
PS2=’>‘
PS4=’+‘
PWD=’/home/atest’
SESSION_MANAGER=‘local/testbox:/tmp/.ICE-unix/5051’
SHELL=’/bin/dash’
SHLVL=‘1’
TERM=‘xterm’
USER=‘atest’
XCURSOR_THEME=‘default’
_=‘ash’
$
Your default dash shell environment will most likely differ, because different Linux
distributionsassigndifferentdefaultenvironmentvariablesatlogin.
Positionalparameters
In addition to the default environment variables, the dash shell also assigns special
variables to any parameters defined in the command line. Here are the positional
parametervariablesavailableforuseinthedashshell:
$0:Thenameoftheshell
$n:Thenthpositionparameter
$*:Asinglevaluewiththecontentsofalltheparameters,separatedbythefirst
characterintheIFSenvironmentvariable,oraspaceifIFSisn’tdefined
$@:Expandstomultipleargumentsconsistingofallthecommandlineparameters
$#:Thenumberofpositionalparameters
$?:Theexitstatusofthemostrecentcommand
$-:Thecurrentoptionflags
$$:TheprocessID(PID)ofthecurrentshell
$!:TheprocessID(PID)ofthemostrecentbackgroundcommand
Allthedashpositionalparametersmimicthesamepositionalparametersavailableinthe
bashshell.Youcanuseeachofthepositionalparametersinyourshellscriptsjustasyou
wouldinthebashshell.
User-definedenvironmentvariables
Thedashshellalsoallowsyoutosetyourownenvironmentvariables.Aswithbash,you
can define a new environment variable on the command line by using the assignment
statement:
$testing=10;exporttesting
$echo$testing
10
$
Withoutthe exportcommand,user-definedenvironmentvariablesarevisibleonlyinthe
currentshellorprocess.
Caution
There’sonehugedifferencebetweendashvariablesandbashvariables.Thedash
shelldoesn’tsupportvariablearrays.Thissmallfeaturecausesallsortsofproblems
foradvancedshellscriptwriters.
Thedashbuilt-incommands
Just as with the bash shell, the dash shell contains a set of built-in commands that it
recognizes.Youcanusethesecommandsdirectlyfromthecommandlineinterface,oryou
can incorporate them in your shell scripts. Table 23.2 lists the dash shell built-in
commands.
Table23.2ThedashShellBuilt-InCommands
Command
Description
alias
Createsanaliasstringtorepresentatextstring
bg
Continuesspecifiedjobinbackgroundmode
cd
Switchestothespecifieddirectory
echo
Displaysatextstringandenvironmentvariables
eval
Concatenatesallargumentswithaspace
exec
Replacestheshellprocesswiththespecifiedcommand
exit
Terminatestheshellprocess
export
Exportsthespecifiedenvironmentvariableforuseinallchildshells
fg
Continuesspecifiedjobinforegroundmode
getopts
Obtainsoptionsandargumentsfromalistofparameters
hash
Maintainsandretrievesahashtableofrecentcommandsandtheirlocations
pwd
Displaysthevalueofthecurrentworkingdirectory
read
ReadsalinefromSTDINandassignthevaluetoavariable
readonly
ReadsalinefromSTDINtoavariablethatcan’tbechanged
printf
Displaystextandvariablesusingaformattedstring
set
Listsorsetsoptionflagsandenvironmentvariables
shift
Shiftsthepositionalparametersaspecifiednumberoftimes
test
Evaluatesanexpressionandreturns0iftrueor1iffalse
Displaystheaccumulateduserandsystemtimesfortheshellandallshell
times
processes
trap
Parsesandexecutesanactionwhentheshellreceivesaspecifiedsignal
Interpretsthespecifiednameanddisplaystheresolution(alias,built-in,
type
command,keyword)
ulimit
umask
unalias
unset
wait
Queriesorsetslimitsonprocesses
Setsthevalueofthedefaultfileanddirectorypermissions
Removesthespecifiedalias
Removesthespecifiedvariableoroptionflagfromtheexportedvariables
Waitsforthespecifiedjobtocompleteandreturnstheexitstatus
Youprobablyrecognizeallthesebuilt-incommandsfromthebashshell.Thedashshell
supportsmanyofthesamebuilt-incommandsasthebashshell.You’llnoticethatthere
arenocommandsforthecommandhistoryfileorforthedirectorystack.Thedashshell
doesn’tsupportthesefeatures.
Scriptingindash
Unfortunately,thedashshelldoesn’trecognizeallthescriptingfeaturesofthebashshell.
Shell scripts written for the bash environment often fail when run in the dash shell,
causing all sorts of grief for shell script programmers. This section describes the
differencesyou’llneedtobeawareoftogetyourshellscriptstorunproperlyinadash
shellenvironment.
Creatingdashscripts
Youprobablyguessedbynowthatcreatingshellscriptsforthedashshellisprettysimilar
tocreatingshellscriptsforthebashshell.Youshouldalwaysspecifywhichshellyouwant
touseinyourscripttoensurethatthescriptrunswiththepropershell.
Youdothisonthefirstlineoftheshell:
#!/bin/dash
You can also specify a shell command line parameter on this line, as was documented
earlierin“Thedashcommandlineparameters”section.
Thingsthatdon’twork
Unfortunately,becausethedashshellisonlyasubsetoftheBourneshellfeatures,some
thingsinbashshellscriptsdon’tworkinthedashshell.Theseareoftencalledbashisms.
Thissectionisaquicksummaryofbashshellfeaturesyoumaybeusedtousinginyour
bashshellscriptsthatdon’tworkifyou’reinadashshellenvironment.
Usingarithmetic
Chapter11showedthreewaystoexpressamathematicaloperationinthebashshellscript:
Usingtheexprcommand:exproperation
Usingsquarebrackets:$[operation]
Usingdoubleparentheses:$((operation))
Thedashshellsupportstheexprcommandandthedoubleparenthesesmethodbutdoesn’t
supportthesquarebracketmethod.Thiscanbeaproblemifyouhavelotsofmathematical
operationsthatusethesquarebrackets.
Theproperformatforperformingmathematicaloperationsindashshellscriptsistouse
thedoubleparenthesesmethod:
$cattest5b
#!/bin/dash
#testingmathematicaloperations
value1=10
value2=15
value3=$(($value1*$value2))
echo“Theansweris$value3”
$./test5b
Theansweris150
$
Nowtheshellcanperformthecalculationpr