Showing posts with label Addison-Wesley. Show all posts

T. J. Klevinsky, Scott Laliberte, Ajay Gupta

Hack I.T.introduces penetration testing and its vital role in an overall network security plan.

Penetration testing--in which professional, "white hat" hackers attempt to break through an
organization’s security defenses--has become a key defense weapon in today’s information systems
security arsenal. Through penetration testing, I.T. and security professionals can take action to prevent
true "black hat" hackers from compromising systems and exploiting proprietary information.

e-books shop
Hack IT Security Through Penetration Testing

T.J. Klevinsky, CISSP
T.J. is a manager with Ernst & Young's Security and Technology Solutions practice. He is
currently responsible for coordinating attack and penetration exercises in various parts of
the world. As an instructor for his company's “Extreme Hacking” course, T.J. is constantly
researching new tools and techniques for exploiting security vulnerabilities. To keep the
course up-to-date, new tools and methods are included in the attack and penetration
methodology. Additionally, as the author and instructor for the System Administration and
Network Security (SANS) Institute course “Contemporary Hacking Tools and Penetration
Testing,” T.J. has had the opportunity to interact with other penetration-testing
professionals across the globe to identify new tools and techniques and to bring these
experiences and tools to this book.

Scott Laliberte
Scott is a manager with Ernst & Young's Security and Technology Solutions practice. He
has extensive experience and expertise in the areas of information systems security,
network operations, and electronic commerce. Specifically, Scott has managed and led
numerous attack and penetration engagements and systems vulnerability assessments for
midsize and Fortune 500 companies. During these engagements Scott used a variety of
commercial and proprietary tools and techniques to identify vulnerabilities in networks,
operating systems, and applications. Scott is also responsible for coordinating and
designing e-commerce architectures and verifying security controls and the effectiveness
of the architectures. In addition, Scott is an instructor for Ernst & Young's “Extreme
Hacking” course, where he helps train others in Ernst & Young's attack and penetration methodology.

Ajay Gupta
Ajay is a senior security professional with Ernst & Young's Security and Technology
Solutions practice, where he performs security reviews for Ernst & Young clients. He has
experience in performing penetration testing, risk analysis, and code review engagements
as well as evaluating the security posture of client organizations ranging from Fortune 100
firms to e-commerce start-ups. Ajay is an instructor for Ernst & Young's “Extreme Hacking”
course and spends a large portion of his time developing and reviewing new tools. Ajay is
one of Ernst & Young's specialists in intrusion detection systems and has evaluated,
installed, and configured various intrusion detection tools. He has been a speaker in the
fields of security and electronic commerce for various national organizations and universities.

Why write a book about hacking? The question is really whether a book about the
techniques and tools used to break into a network would be beneficial to the information
security community. We, the authors, believe that penetration testing is a valuable and
effective means of identifying security holes and weaknesses in a network and computing
environment. Understanding how others will try to break into a network offers considerable
insight into the common pitfalls and misconfigurations that make networks vulnerable. This
insight is essential to creating a comprehensive network security structure.

Some may argue that providing this penetration-testing information gives script kiddies and
hackers ammunition to better attack systems. However, script kiddies and hackers already
have access to this information or have the time to find it—most of the material presented
in this book is available from a variety of sources on the Internet. The problem is that the
system and security administrators defending against attacks do not have the time or
resources to research the sites necessary to compile this information. We decided to write
this book to provide defenders with the information hackers already have. A hacker has to
find only one hole to gain unauthorized access. The security group defending against the
hackers needs to find all the holes to prevent unauthorized access.

There is no tried-and-true training that can make everyone a security expert, but there are
some baseline principles, skills, and tools that must be mastered to become proficient in
this field. Our goal is to provide you with those skills in a manner that helps you to
understand the structure and tools used and to begin developing your own style of penetration testing.

The process described in this book is not the only way to perform a penetration test. We
continue to evolve our own methodology to respond to new technologies and threats. This
process has worked well for us in the past and continues to be a successful way to
evaluate and test network security.


It certainly seems that over the past few years the security ramifications of online activity
have begun to permeate the national consciousness. Mainstream media have begun to
take an interest in and glamorize the compromises that have taken place. Even Hollywood
has movies about hacking, the latest being Warner Brothers' Swordfish starring John
Travolta, Halle Berry, and Hugh Jackman as the world's foremost hacker.

Despite the growing level of interest in this field, there is still little known about the actual
issues involved in securing networks and electronic assets. Many people consider
anti-virus software used to defend against Internet e-mail viruses to be the cure-all for all
varieties of information security threats. Viruses are a big problem, no doubt, potentially
leading to huge losses in terms of lost productivity and corrupted intellectual assets.
However, cyber crime (hacking) can be much more than the release of an e-mail
attachment that proclaims love (the I LOVE YOU virus) or promises sexy pictures (the
Anna Kournikova virus) to all the friends and business associates of unsuspecting victims.
The true dangers of cyber crime are of far greater consequence. Individuals with technical
knowledge of networks and networking devices can steal sensitive information (for
example, U.S. troop deployments from Department of Defense computers, source code for
new software products, medical records) or money (through online access to bank
accounts or credit card numbers used with online retailers) or conduct a host of juvenile
pranks (erasing backup files recording the last six months of activity, raising the
temperature in buildings, turning off phone systems).

While these may seem to be scare tactics used to get people to spend time, energy, and
good money on unnecessary things, that is, unfortunately, not the case. The threats are
real. They are evident in the latest “Computer Crime and Security Survey” by the Computer
Security Institute and the Federal Bureau of Investigation and in news reports of cases of
identity theft and firms facing the realization that they are being blackmailed by a hacker
who has their customer list (including credit card information).
Given this burgeoning interest in keeping networks free from hacking minds, there has
naturally been greater interest in taking steps to ensure networks are secure. One such
step is to perform a professional penetration test, also called attack and penetration or
ethical hacking. There are various parts of the security industry, namely those people who
provide security consulting services (also called professional services), those who develop
and market security products, and finally those who are managed security service
providers (MSSPs).

MSSPs provide outsourced security monitoring and management of all or parts of a
network in exchange for a retainer. Firewalls, intrusion detection systems, audit logs, and
virus scanners can all be managed by an MSSP. The developers of security products
include commercial interests, a large open-source community, and smaller groups of black
hat hackers who aim to create tools to automate the network analysis and review process.
Such tools include firewalls, intrusion detection systems, auditing tools, virus scanners,
vulnerability scanners, network mappers, network sniffers, encryption tools, password
crackers, banner grabbers … the list goes on. In addition, tools and scripts, such as
denial-of-service exploits, that aid in the compromise of networks are also frequently
developed and released. Naturally, this later set of tools come generally from the domain of
open-source or black hat developers, while commercial interests stick to more benign offerings.

Penetration-testing services are a component of consulting services. Consulting services
also include the development of security policies and procedures, the performance of
security vulnerability and risk analysis of networks, and the design and implementation of
security solutions (such as a firewall solution, a public key infrastructure, a single sign-on
solution, or an IDS solution) and a host of related services. The goal of security consulting
services, especially for penetration testing, is to improve or augment the security posture of
a network or system.

“And he that breaks a thing to find out what it is has left the path of wisdom.”
—Galdalf the Grey from The Fellowship of the Ring, Volume 1 of The Lord
of the Rings by J.R.R. Tolkien
This sentiment applies to penetration testing. Our testing does not intend to and never
should actually cripple or compromise a network. However, testing must detect as many
ways to do so as possible. The findings or results of the testing are aimed at improving the
security posture of a network by presenting countermeasures for the vulnerabilities
identified. The process is simple: take a few white hat hackers, give them black hats for a
short period of time, and let them try to figure out all the possible ways a system can be
compromised. Then, take the black hats away and have them report on their findings—to
the client, not to the general Internet hacker community.

This book focuses on presenting a method for performing penetration testing. In doing so,
we do not discuss other consulting services available. And while we do discuss in some
detail the tools we use for penetration testing, this work should not be considered a
comprehensive review of the security products available in the market today. We also do
not address the burgeoning MSSP field, though we briefly discuss it in the final chapter on future trends.

We, the authors, share a connection with the professional services firm Ernst & Young
LLP. We attest that the ideas and opinions presented throughout this work are not
necessarily those of Ernst & Young but solely the critical analysis based on our years of field experience.

Truth be told, much of the information presented here can be found in various places on
the Web, in news groups, in e-mail distribution lists, or at other destinations on the Internet
(a listing is presented in Chapter 22). Those who believe writing such a book is dangerous
since it may result in teaching people how to hack do not see the value in improving
security through testing and measuring defenses against the techniques of opponents.
Hackers already know how to hack and have the time and energy to research (and
develop) hacking techniques. The good guys, who are busy battling the day-to-day fires of
maintaining the corporate network, do not have the luxury of this time and cannot perform
this level of research. We hope this book will be a tool for the good guys. It consolidates
and organizes the information already available to the hacker community so that security
professionals can arm themselves in the security battle.

We hope you find this text as useful to read as it was challenging for us to write. We are
glad to provide our knowledge and intelligence on penetration testing. How you choose to
use it is of your own volition. Remember: Penetration testing without permission is
illegal—a point we hope this text makes clear.
Happy reading.


e-books shop

Purchase Now !
Just with Paypal

Product details
 File Size
 12,564 KB
 575 p
 File Type
 PDF format
 2002 by Pearson Education, Inc 

Hack I.T.: 
Security Through Penetration Testing

How to Use This Book
1. Hacking Today
2. Defining the Hacker
2.1 Hacker Skill Levels
2.2 Information Security Consultants
2.3 Hacker Myths
2.4 Information Security Myths
3. Penetration for Hire
3.1 Ramifications of Penetration Testing
3.2 Requirements for a Freelance Consultant
3.3 Announced vs. Unannounced Penetration Testing
4. Where the Exposures Lie
4.1 Application Holes
4.2 Berkeley Internet Name Domain ( BIND ) Implementations
4.3 Common Gateway Interface ( CGI )
4.4 Clear Text Services
4.5 Default Accounts
4.6 Domain Name Service ( DNS )
4.7 File Permissions
4.8 FTP and telnet
4.9 ICMP
4.10 IMAP and POP
4.11 Modems
4.12 Lack of Monitoring and Intrusion Detection
4.13 Network Architecture
4.14 Network File System ( NFS )
4.15 NT Ports 135?139
4.16 NT Null Connection
4.17 Poor Passwords and User IDs
4.18 Remote Administration Services
4.19 Remote Procedure Call ( RPC )
4.21 Services Started by Default
4.22 Simple Mail Transport Protocol ( SMTP )
4.23 Simple Network Management Protocol ( SNMP ) Community Strings
4.24 Viruses and Hidden Code
4.25 Web Server Sample Files
4.26 Web Server General Vulnerabilities
4.27 Monitoring Vulnerabilities
5. Internet Penetration
5.1 Network Enumeration/Discovery
5.2 Vulnerability Analysis
5.3 Exploitation
Case Study: Dual-Homed Hosts
6. Dial-In Penetration
6.1 War Dialing
6.2 War Dialing Method
6.3 Gathering Numbers
6.4 Precautionary Methods
6.5 War Dialing Tools
Case Study: War Dialing
7. Testing Internal Penetration
7.1 Scenarios
7.2 Network Discovery
7.3 NT Enumeration
7.4 UNIX
7.5 Searching for Exploits
7.6 Sniffing
7.7 Remotely Installing a Hacker Tool Kit
7.8 Vulnerability Scanning
Case Study: Snoop the User Desktop
8. Social Engineering
8.1 The Telephone
8.2 Dumpster Diving
8.3 Desktop Information
8.4 Common Countermeasures
9. UNIX Methods
9.1 UNIX Services
9.2 Buffer Overflow Attacks
9.3 File Permissions
9.4 Applications
9.5 Misconfigurations
9.6 UNIX Tools
Case Study: UNIX Penetration
10. The Tool Kit
10.1 Hardware
10.2 Software
10.3 VMware
11. Automated Vulnerability Scanners
11.1 Definition
11.2 Testing Use
11.3 Shortfalls
11.4 Network-Based and Host-Based Scanners
11.5 Tools
11.6 Network-Based Scanners
11.7 Host-Based Scanners
11.8 Pentasafe VigilEnt
11.9 Conclusion
12. Discovery Tools
12.1 WS_Ping ProPack
12.2 NetScanTools
12.3 Sam Spade
12.4 Rhino9 Pinger
12.5 VisualRoute
12.6 Nmap
12.7 What's running
13. Port Scanners
13.1 Nmap
13.2 7th Sphere Port Scanner
13.3 Strobe
13.4 SuperScan
14. Sniffers
14.1 Dsniff
14.2 Linsniff
14.3 Tcpdump
14.4 BUTTSniffer
14.5 SessionWall-3 (Now eTrust Intrusion Detection)
14.6 AntiSniff
15. Password Crackers
15.1 L0phtCrack
15.2 pwdump2
15.3 John the Ripper
15.4 Cain
15.5 ShowPass
16. Windows NT Tools
16.1 NET USE
16.2 Null Connection
16.6 epdump
16.8 Getmac
16.9 Local Administrators
16.10 Global (?Domain Admins?)
16.11 Usrstat
16.12 DumpSec
16.13 user2Sid/sid2User
16.14 NetBIOS Auditing Tool ( NAT )
16.15 SMBGrind
16.18 AuditPol
16.19 REGDMP
16.20 Somarsoft DumpReg
16.21 Remote
16.22 Netcat
16.23 SC
16.24 AT
16.25 FPipe
Case Study: Weak Passwords
Case Study: Internal Penetration to Windows
17. Web-Testing Tools
17.1 Whisker
17.2 SiteScan
17.3 THC Happy Browser
17.4 wwwhack
17.5 Web Cracker
17.6 Brutus
Case Study: Compaq Management Agents Vulnerability
18. Remote Control
18.1 pcAnywhere
18.2 Virtual Network Computing
18.3 NetBus
18.4 Back Orifice 2000
19. Intrusion Detection Systems
19.1 Definition
19.2 IDS Evasion
19.3 Pitfalls
19.4 Traits of Effective IDSs
19.5 IDS Selection
20. Firewalls
20.1 Definition
20.2 Monitoring
20.3 Configuration
20.4 Change Control
20.5 Firewall Types
20.6 Network Address Translation
20.7 Evasive Techniques
20.8 Firewalls and Virtual Private Networks
Case Study: Internet Information Server Exploit?MDAC
21. Denial-of-Service Attacks
21.1 Resource Exhaustion Attacks
21.2 Port Flooding
21.3 SYN Flooding
21.4 IP Fragmentation Attacks
21.5 Distributed Denial-of-Service Attacks
21.6 Application-Based DoS Attacks
21.7 Concatenated DoS Tools
21.8 Summary
22. Wrapping It Up
22.1 Countermeasures
22.2 Keeping Current
23. Future Trends
23.1 Authentication
23.2 Encryption
23.3 Public Key Infrastructure
23.4 Distributed Systems
23.5 Forensics
23.6 Government Regulation
23.7 Hacking Techniques
23.8 Countermeasures
23.9 Cyber-Crime Insurance
A. CD-ROM Contents
Organization of the CD-ROM
Compilation of Programs
B. The Twenty Most Critical Internet Security Vulnerabilities?The Experts'
The SANS Institute
G1?Default Installs of Operating Systems and Applications
G2?Accounts with No Passwords or Weak Passwords
G3?Non-existent or Incomplete Backups
G4?Large Number of Open Ports
G5?Not Filtering Packets for Correct Incoming and Outgoing Addresses
G6?Non-existent or Incomplete Logging
G7?Vulnerable CGI Programs
W1? Unicode Vulnerability (Web Server Folder Traversal)
W2?ISAPI Extension Buffer Overflows
W3? IIS RDS Exploit (Microsoft Remote Data Services)
W4?NETBIOS?Unprotected Windows Networking Shares
W5?Information Leakage Via Null Session Connections
W6?Weak Hashing in SAM ( LM Hash)
U1?Buffer Overflows in RPC Services
U2?Sendmail Vulnerabilities
U3?Bind Weaknesses
U4?R Commands
U5?LPD (Remote Print Protocol Daemon)
U6?Sadmind and Mountd
U7?Default SNMP Strings
Appendix Appendix A ?Common Vulnerable Ports
Appendix Appendix B ?The Experts Who Helped Create the Top Ten and Top
Twenty Internet Vulnerability list

═════ ═════

David M. Beazley

Contents at a Glance

Part I: The Python Language
A Tutorial Introduction
Lexical Conventions and Syntax
Types and Objects
Operators and Expressions
Program Structure and Control Flow
Functions and Functional Programming
Classes and Object-Oriented Programming
Modules, Packages, and Distribution
Input and Output
Execution Environment
Testing, Debugging, Profiling, and Tuning
Part II: The Python Library
Built-In Functions
Python Runtime Services
Data Structures, Algorithms, and Code Simplification
String and Text Handling
Python Database Access
File and Directory Handling
Operating System Services
Threads and Concurrency
Network Programming and Sockets
Internet Application Programming
Web Programming 
Internet Data Handling and Encoding
Miscellaneous Library Modules
Part III: Extending and Embedding
Extending and Embedding Python 
Appendix: Python 3

e-books shop
e-books shop
Purchase Now !
Just with Paypal

Book Details
 745 p
 File Size 
 3,580 KB
 File Type
 PDF format
 2009 by Pearson Education, Inc 

About the Author
David M. Beazley is a long-time Python enthusiast, having been involved with the
Python community since 1996. He is probably best known for his work on SWIG, a
popular software package for integrating C/C++ programs with other programming
languages, including Python, Perl, Ruby,Tcl, and Java. He has also written a number of
other programming tools, including PLY, a Python implementation of lex and yacc.
Dave spent seven years working in the Theoretical Physics Division at Los Alamos
National Laboratory, where he helped pioneer the use of Python with massively parallel
supercomputers. After that, Dave went off to work as an evil professor, where he
enjoyed tormenting college students with a variety of insane programming projects.
However, he has since seen the error of his ways and is now working as an independent
software developer, consultant, Python trainer, and occasional jazz musician living in
Chicago. He can be contacted at

About the Technical Editor
Noah Gift is the co-author of Python For UNIX and Linux System Administration
(O’Reilly) and is also working on Google App Engine In Action (Manning). He is an
author, speaker, consultant, and community leader, writing for publications such as IBM
developerWorks, Red Hat Magazine, O’Reilly, and MacTech. His consulting company’s
website is, and much of his writing can be found at can also follow Noah on Twitter.
Noah has a master’s degree in CIS from Cal State, Los Angeles, a B.S. in nutritional
science from Cal Poly San Luis Obispo, is an Apple and LPI-certified SysAdmin, and
has worked at companies such as Caltech, Disney Feature Animation, Sony Imageworks,
and Turner Studios. He is currently working at Weta Digital in New Zealand. In his free
time he enjoys spending time with his wife Leah and their son Liam, composing for the
piano, running marathons, and exercising religiously.

Table of Contents
Introduction 1
I: The Python Language
1 A Tutorial Introduction 5
Running Python 5
Variables and Arithmetic Expressions 7
Conditionals 9
File Input and Output 10
Strings 11
Lists 12
Tuples 14
Sets 15
Dictionaries 16
Iteration and Looping 17
Functions 18
Generators 19
Coroutines 20
Objects and Classes 21
Exceptions 22
Modules 23
Getting Help 24
2 Lexical Conventions and Syntax 25
Line Structure and Indentation 25
Identifiers and Reserved Words 26
Numeric Literals 26
String Literals 27
Containers 29
Operators, Delimiters, and Special Symbols 30
Documentation Strings 30
Decorators 30
Source Code Encoding 31
3 Types and Objects 33
Terminology 33
Object Identity and Type 33
Reference Counting and Garbage Collection 34
References and Copies 35
From the Library of Lee Bogdanoff
viii Contents
First-Class Objects 36
Built-in Types for Representing Data 37
The None Type 38
Numeric Types 38
Sequence Types 39
Mapping Types 44
Set Types 46
Built-in Types for Representing Program Structure 47
Callable Types 47
Classes, Types, and Instances 50
Modules 50
Built-in Types for Interpreter Internals 51
Code Objects 51
Frame Objects 52
Traceback Objects 52
Generator Objects 53
Slice Objects 53
Ellipsis Object 54
Object Behavior and Special Methods 54
Object Creation and Destruction 54
Object String Representation 55
Object Comparison and Ordering 56
Type Checking 57
Attribute Access 57
Attribute Wrapping and Descriptors 58
Sequence and Mapping Methods 58
Iteration 59
Mathematical Operations 60
Callable Interface 62
Context Management Protocol 62
Object Inspection and dir() 63
4 Operators and Expressions 65
Operations on Numbers 65
Operations on Sequences 67
String Formatting 70
Advanced String Formatting 72
Operations on Dictionaries 74
Operations on Sets 75
Augmented Assignment 75
From the Library of Lee Bogdanoff
Contents ix
The Attribute (.) Operator 76
The Function Call () Operator 76
Conversion Functions 76
Boolean Expressions and Truth Values 77
Object Equality and Identity 78
Order of Evaluation 78
Conditional Expressions 79
5 Program Structure and Control Flow 81
Program Structure and Execution 81
Conditional Execution 81
Loops and Iteration 82
Exceptions 84
Built-in Exceptions 86
Defining New Exceptions 88
Context Managers and the with Statement 89
Assertions and __debug__ 91
6 Functions and Functional Programming 93
Functions 93
Parameter Passing and Return Values 95
Scoping Rules 96
Functions as Objects and Closures 98
Decorators 101
Generators and yield 102
Coroutines and yield Expressions 104
Using Generators and Coroutines 106
List Comprehensions 108
Generator Expressions 109
Declarative Programming 110
The lambda Operator 112
Recursion 112
Documentation Strings 113
Function Attributes 114
eval(), exec(), and compile() 115
7 Classes and Object-Oriented Programming 117
The class Statement 117
Class Instances 118
Scoping Rules 118
Inheritance 119
From the Library of Lee Bogdanoff
Polymorphism Dynamic Binding and Duck Typing 122
Static Methods and Class Methods 123
Properties 124
Descriptors 126
Data Encapsulation and Private Attributes 127
Object Memory Management 128
Object Representation and Attribute Binding 131
__slots__ 132
Operator Overloading 133
Types and Class Membership Tests 134
Abstract Base Classes 136
Metaclasses 138
Class Decorators 141
8 Modules, Packages, and Distribution 143
Modules and the import Statement 143
Importing Selected Symbols from a Module 145
Execution as the Main Program 146
The Module Search Path 147
Module Loading and Compilation 147
Module Reloading and Unloading 149
Packages 149
Distributing Python Programs and Libraries 152
Installing Third-Party Libraries 154
9 Input and Output 157
Reading Command-Line Options 157
Environment Variables 158
Files and File Objects 158
Standard Input, Output, and Error 161
The print Statement 162
The print() Function 163
Variable Interpolation in Text Output 163
Generating Output 164
Unicode String Handling 165
Unicode I/O 167
Unicode Data Encodings 168
Unicode Character Properties 170
Object Persistence and the pickle Module 171
x Contents
From the Library of Lee Bogdanoff
10 Execution Environment 173
Interpreter Options and Environment 173
Interactive Sessions 175
Launching Python Applications 176
Site Configuration Files 177
Per-user Site Packages 177
Enabling Future Features 178
Program Termination 179
11 Testing, Debugging, Profiling, and Tuning 181
Documentation Strings and the doctest Module 181
Unit Testing and the unittest Module 183
The Python Debugger and the pdb Module 186
Debugger Commands 187
Debugging from the Command Line 189
Configuring the Debugger 190
Program Profiling 190
Tuning and Optimization 191
Making Timing Measurements 191
Making Memory Measurements 192
Disassembly 193
Tuning Strategies 194
II: The Python Library 199
12 Built-In Functions and Exceptions 201
Built-in Functions and Types 201
Built-In Exceptions 212
Exception Base Classes 212
Exception Instances 212
Predefined Exception Classes 213
Built-In Warnings 216
future_builtins 217
13 Python Runtime Services 219
atexit 219
copy 219
Notes 220
Contents xi
From the Library of Lee Bogdanoff
gc 220
Notes 222
inspect 222
marshal 226
Notes 226
pickle 226
Notes 229
sys 229
Variables 229
Functions 233
traceback 235
types 237
Notes 237
warnings 238
Notes 239
weakref 240
Example 242
Notes 242
14 Mathematics 243
decimal 243
Decimal Objects 244
Context Objects 244
Functions and Constants 247
Examples 248
Notes 249
fractions 250
math 251
Notes 252
numbers 252
Notes 253
random 254
Seeding and Initialization 254
Random Integers 254
Random Sequences 254
Real-Valued Random Distributions 255
Notes 256
xii Contents
From the Library of Lee Bogdanoff
15 Data Structures, Algorithms, and Code Simplification 257
abc 257
array 259
Notes 261
bisect 261
collections 262
deque and defaultdict 262
Named Tuples 263
Abstract Base Classes 265
contextlib 267
functools 268
heapq 269
itertools 270
Examples 273
operator 273
16 String and Text Handling 277
codecs 277
Low-Level codecs Interface 277
I/O-Related Functions 279
Useful Constants 279
Standard Encodings 280
Notes 280
re 281
Pattern Syntax 281
Functions 283
Regular Expression Objects 284
Match Objects 285
Example 286
Notes 287
string 287
Constants 287
Formatter Objects 288
Template Strings 289
Utility Functions 290
struct 290
Packing and Unpacking Functions 290
Struct Objects 291
Contents xiii
From the Library of Lee Bogdanoff
Format Codes 291
Notes 292
unicodedata 293
17 Python Database Access 297
Relational Database API Specification 297
Connections 297
Cursors 298
Forming Queries 300
Type Objects 301
Error Handling 302
Multithreading 302
Mapping Results into Dictionaries 303
Database API Extensions 303
sqlite3 Module 303
Module-Level Functions 304
Connection Objects 305
Cursors and Basic Operations 308
DBM-Style Database Modules 310
shelve Module 311
18 File and Directory Handling 313
bz2 313
filecmp 314
fnmatch 316
Examples 316
glob 317
Example 317
gzip 317
Notes 317
shutil 318
tarfile 319
Exceptions 322
Example 322
tempfile 323
zipfile 324
zlib 328
xiv Contents
From the Library of Lee Bogdanoff
19 Operating System Services 331
commands 331
Notes 332
configParser, Configparser 332
The ConfigParser Class 332
Example 334
Notes 336
datetime 336
date Objects 336
time Objects 338
datetime objects 339
timedelta objects 340
Mathematical Operations Involving Dates 341
tzinfo Objects 342
Date and Time Parsing 343
errno 343
POSIX Error Codes 344
Windows Error Codes 346
fcntl 347
Example 348
Notes 349
io 349
Base I/O Interface 349
Raw I/O 350
Buffered Binary I/O 351
Text I/O 353
The open() Function 354
Abstract Base Classes 354
logging 355
Logging Levels 355
Basic Configuration 355
Logger Objects 356
Handler Objects 362
Message Formatting 364
Miscellaneous Utility Functions 366
Logging Configuration 366
Performance Considerations 369
Notes 369
Contents xv
From the Library of Lee Bogdanoff
mmap 369
Notes 372
msvcrt 372
optparse 374
Example 377
Notes 378
os 378
Process Environment 379
File Creation and File Descriptors 381
Files and Directories 386
Process Management 390
System Configuration 395
Exceptions 396
os.path 396
signal 399
Example 401
Notes 401
subprocess 402
Examples 404
Notes 404
time 405
Notes 407
winreg 408
Notes 411
20 Threads and Concurrency 413
Basic Concepts 413
Concurrent Programming and Python 414
multiprocessing 415
Processes 415
Interprocess Communication 417
Process Pools 424
Shared Data and Synchronization 426
Managed Objects 428
Connections 433
Miscellaneous Utility Functions 434
General Advice on Multiprocessing 435
threading 436
Thread Objects 436
Timer Objects 437
xvi Contents
From the Library of Lee Bogdanoff
Lock Objects 438
RLock 438
Semaphore and Bounded Semaphore 439
Events 440
Condition Variables 441
Working with Locks 442
Thread Termination and Suspension 443
Utility Functions 443
The Global Interpreter Lock 444
Programming with Threads 444
queue, Queue 444
Queue Example with Threads 445
Coroutines and Microthreading 446
21 Network Programming and Sockets 449
Network Programming Basics 449
asynchat 452
asyncore 455
Example 457
select 459
Advanced Module Features 460
Advanced Asynchronous I/O Example 460
When to Consider Asynchronous Networking 467
socket 469
Address Families 470
Socket Types 470
Addressing 471
Functions 473
Exceptions 485
Example 485
Notes 486
ssl 486
Examples 489
SocketServer 489
Handlers 490
Servers 491
Defining Customized Servers 492
Customization of Application Servers 494
Contents xvii
From the Library of Lee Bogdanoff
22 Internet Application Programming 497
ftplib 497
Example 500
http Package 500
http.client (httplib) 502
http.server (BaseHTTPServer, CGIHTTPServer,
SimpleHTTPServer) 506
http.cookies (Cookie) 511
http.cookiejar (cookielib) 513
smtplib 513
Example 514
urllib Package 514
urllib.request (urllib2) 515
urllib.response 520
urllib.parse 520
urllib.error 523
urllib.robotparser (robotparser) 523
Notes 524
xmlrpc Package 524
xmlrpc.client (xmlrpclib) 524
xmlrpc.server (SimpleXMLRPCServer, DocXMLRPCServer)
23 Web Programming 531
cgi 533
CGI Programming Advice 537
Notes 538
cgitb 539
wsgiref 540
The WSGI Specification 540
wsgiref Package 542
webbrowser 544
24 Internet Data Handling and Encoding 545
base64 545
binascii 547
csv 548
Dialects 551
Example 551
xviii Contents
From the Library of Lee Bogdanoff
email Package 552
Parsing Email 552
Composing Email 555
Notes 558
hashlib 559
hmac 559
Example 560
HTMLParser 561
Example 562
json 563
mimetypes 566
quopri 567
xml Package 568
XML Example Document 569
xml.dom.minidom 570
xml.etree.ElementTree 573
xml.sax 580
xml.sax.saxutils 583
25 Miscellaneous Library Modules 585
Python Services 585
String Processing 586
Operating System Modules 586
Network 587
Internet Data Handling 587
Internationalization 587
Multimedia Services 588
Miscellaneous 588
III: Extending and Embedding 589
26 Extending and Embedding Python 591
Extension Modules 591
An Extension Module Prototype 593
Naming Extension Modules 595
Compiling and Packaging Extensions 596
Type Conversion from Python to C 597
Type Conversion from C to Python 602
Contents xix
From the Library of Lee Bogdanoff
Adding Values to a Module 604
Error Handling 605
Reference Counting 607
Threads 607
Embedding the Python Interpreter 608
An Embedding Template 608
Compilation and Linking 608
Basic Interpreter Operation and Setup 608
Accessing Python from C 610
Converting Python Objects to C 611
ctypes 612
Loading Shared Libraries 612
Foreign Functions 612
Datatypes 613
Calling Foreign Functions 615
Alternative Type Construction Methods 616
Utility Functions 617
Example 618
Advanced Extending and Embedding 619
Jython and IronPython 620
Appendix Python 3 621
Who Should Be Using Python 3? 621
New Language Features 622
Source Code Encoding and Identifiers 622
Set Literals 622
Set and Dictionary Comprehensions 623
Extended Iterable Unpacking 623
Nonlocal Variables 624
Function Annotations 624
Keyword-Only Arguments 625
Ellipsis as an Expression 626
Chained Exceptions 626
Improved super() 627
Advanced Metaclasses 627
Common Pitfalls 629
Text Versus Bytes 629
New I/O System 631
xx Contents
From the Library of Lee Bogdanoff
print() and exec() Functions 631
Use of Iterators and Views 632
Integers and Integer Division 633
Comparisons 633
Iterators and Generators 633
File Names, Arguments, and Environment Variables 633
Library Reorganization 634
Absolute Imports 634
Code Migration and 2to3 634
Porting Code to Python 2.6 634
Providing Test Coverage 635
Using the 2to3 Tool 635
A Practical Porting Strategy 637
Simultaneous Python 2 and Python 3 Support 638
Participate 638
Index 639
Contents xxi

e-books shop

This book is intended to be a concise reference to the Python programming language.
Although an experienced programmer will probably be able to learn Python from this
book, it’s not intended to be an extended tutorial or a treatise on how to program.
Rather, the goal is to present the core Python language, and the most essential parts of
the Python library in a manner that’s accurate and concise.This book assumes that the
reader has prior programming experience with Python or another language such as C
or Java. In addition, a general familiarity with systems programming topics (for example,
basic operating system concepts and network programming) may be useful in understanding
certain parts of the library reference.
Python is freely available for download at are available
for almost every operating system, including UNIX,Windows, and Macintosh. In
addition, the Python website includes links to documentation, how-to guides, and a
wide assortment of third-party software.

This edition of Python Essential Reference comes at a pivotal time in Python’s evolution.
Python 2.6 and Python 3.0 are being released almost simultaneously.Yet, Python 3
is a release that breaks backwards compatibility with prior Python versions. As an author
and programmer, I’m faced with a dilemma: do I simply jump forward to Python 3.0 or
do I build upon the Python 2.x releases that are more familiar to most programmers?
Years ago, as a C programmer I used to treat certain books as the ultimate authority
on what programming language features should be used. For example, if you were using
something that wasn’t documented in the K&R book, it probably wasn’t going to be
portable and should be approached with caution.This approach served me very well as
a programmer and it’s the approach I have decided to take in this edition of the
Essential Reference. Namely, I have chosen to omit features of Python 2 that have been
removed from Python 3. Likewise, I don’t focus on features of Python 3 that have not
been back-ported (although such features are still covered in an appendix). As a result, I
hope this book can be a useful companion for Python programmers, regardless of what
Python version is being used.

The fourth edition of Python Essential Reference also includes some of the most exciting
changes since its initial publication nearly ten years ago. Much of Python’s development
throughout the last few years has focused on new programming language features—
especially related to functional and meta programming. As a result, the chapters
on functions and object-oriented programming have been greatly expanded to cover
topics such as generators, iterators, coroutines, decorators, and metaclasses.The library
chapters have been updated to focus on more modern modules. Examples and code
fragments have also been updated throughout the book. I think most programmers will
be quite pleased with the expanded coverage.

Finally, it should be noted that Python already includes thousands of pages of useful
documentation.The contents of this book are largely based on that documentation, but
with a number of key differences. First, this reference presents information in a much
more compact form, with different examples and alternative descriptions of many topics.
Second, a significant number of topics in the library reference have been expanded
From the Library of Lee Bogdanoff
to include outside reference material.This is especially true for low-level system and
networking modules in which effective use of a module normally relies on a myriad of
options listed in manuals and outside references. In addition, in order to produce a more
concise reference, a number of deprecated and relatively obscure library modules have
been omitted.
In writing this book, it has been my goal to produce a reference containing virtually
everything I have needed to use Python and its large collection of modules. Although
this is by no means a gentle introduction to the Python language, I hope that you find
the contents of this book to be a useful addition to your programming reference library
for many years to come. I welcome your comments.
David Beazley
Chicago, Illinois
June, 2009
Loading... Protection Status