Wednesday, December 7, 2011

Build a Linux PXE server to provision Linux and Windows Servers

Windows Deployment Service (WDS)/SCCM is the traditional way to provision  Windows servers, However if you need to provision both Linux and Windows Servers, it would be nice to have a all-in-one server. Before Windows Preinstallation Environment (WinPE) was developed, disk imaging tool like ghost is used to deploy syspreped windows OS as non-windows based provision solution. WinPE is part of Windows Automated Installation Kit(WAIK), it includes, imagex, a tool to capture image on file system level rather than raw sector level.

The PXE Server presented here has the following features:

- Support any PXE capable Linux distribution and most Windows versions: 2000/2003/XP/Vista/7/2008/2008 R2. Potentially, support any other PXE capable server e.g Solaris
- Zero touch installation/full automation
- Dynamic loading configuration using CGI script of your choice PHP/Perl ...
- Downloading kernel/images over http which is more reliable and faster than tftp, additionally, http makes it possible to provision over remote site

Environment setup

7 VMS in Virtualbox 4( for testing provision Centos 5.5, Windows 2003, and Windows 2008 R2)
- PXE server: Centos 5.5
- PXE Linux Client: Centos 5.5
- WAIK tools computer: Windows 2008 R2
- PXE Windows Client(imagex/sysprep target): Windows 2008 R2
- PXE Windows Client(imagex/sysprep target): Windows 2003 R2
- Imagex/sysprep source: Windows 2008 R2
- Imagex/sysprep source: Windows 2003 R2
- Select PCnet-fast III network adapter to support PXE boot
- Windows VM must have at least 1G RAM, because WinPE ISO memdisk will consume few hundreds of MB
- Setup samba share in PXEServer to store Windows imagex(.wim ) files,autoscript for disk partitioning/apply .wim image etc(autorun.bat).
Refer to
Just the filename parameter need to be set to:  filename "gpxelinux.0";
Install pxeserver(syslinux)
The original syslinux package in Centos 5.5 doesn't support memdisk, I use syslinux-4.03
Upgrade to syslinux-4.03 with RPM binary package, but the gpxelinux.0 need to customized and compiled from source.
gPXE/pxelinux backgroud
gPXE( or it is successor iPXE ( support dynamic configuration using any CGI script and support kernel/images loading over http.
Because gPXE/iPXE doesn't support memdisk, which is essential for loading winPE ISO image, we have to use pxelinux in syslinux. Fortunately, syslinux integrates gPXE, so we can switch between gPXE and pxelinux freely using CGI script. That is, use gPXE for Linux, use pxelinux for Windows
Compile customized gPXE in syslinux source file
$cd ${syslinux source}/gpxe/ 
$vi pxelinux.gpxe
set use-cached 0
dhcp net0
chain http://pxe-server/boot/home.php?mac=${net0/mac}
# "!gpxe" is interpreter, not a comment line 
# chain http://pxe-server/boot/home.php?mac=${net0/mac}?will pass the MAC address to a CGI script, which can generate individual configuration based on MAC address
#To use the DNS name, pxe-server, a working DNS server need to be advertised to DHCP client.
$cp gpxelinux.0 /tftpboot/
How does the CGI script work?

A CGI script, php in this example, retrieve the MAC address passed by gPXE request, lookup host-os mapping table, then generate PXE configuration dynamically.

#The configuration file to define which OS to apply based on MAC address
$cat ${wwwroot}/boot/hosts/host-os.conf 
#Mac               os
08:00:27:4d:4d:aa, centos-5.5-i386
08:00:27:b2:40:3e, windows-2008r2-x64

#Linux Server can use gPXE directly
$curl http://pxe-server/boot/home.php?mac=08:00:27:4d:4d:aa
kernel http://pxe-server/boot/os/centos-5.5-i386/pxeboot/vmlinuz ksdevice=link  ks=http://pxe-server/boot/hosts/linux-ks/08-00-27-4d-4d-aa.txt
initrd http://pxe-server/boot/os/centos-5.5-i386/pxeboot/initrd.img 

#Windows server has to rely on pxelinux to support memdisk(part of syslinux package)
$curl http://pxe-server/boot/home.php?mac=08:00:27:b2:40:3e
imgload pxelinux.0
boot pxelinux.0

#A pxelinux config is generated in the mean time 
$cat  /tftpboot/pxelinux.cfg/01-08-00-27-b2-40-3e 
default pe
label  pe
kernel http://pxe-server/boot/os/memdisk
append iso
initrd http://pxe-server/boot/os/w2k8_pe_x64.iso
Implementing Zero touch installation/full automation

Booting up the server via PXE is the first phrase of zero touch installation, OS level automation is done by Linux unattended install: e.g. Kickstart for Redhat Linux family or Windows unattended install:syspreped windows, startnet.cmd in winPE, autoscript in a network share
The following focus on Windows unattended install.
Download WAIK for Windows 7/Windows 2008 R2
How can you use WAIK for Windows 2008 R2 to deploy Windows2000/xp/2003
 It is true that you can't run setup.exe of Windows 2003 in 2008 R2 winPE, but we use imagex tool to capture files, so it works on most Windows OS.
Create winPE ISO
Install the WAIK  in WAIK tools computer
Steps to create winPE ISO
#Open the deployment tools command prompt
#amd64 is for 64bit, use x86 for 32bit
$\>copype.cmd amd64 C:\winpe_amd64
####Optional:Mount the wim file in order to add auto deploy script, you can skip this step to make it easier to get it going.
####BEGIN script customization
$\>imagex /mountrw c:\winpe_amd64\winpe.wim 1 c:\winpe_amd64\mount
#modify C:\winpe_amd64\mount\Windows\System32\starnet.cmd, add following lines below 'wpeinit'
REM -- Start
REM Mount Linux samba share
net use J: \\pxe-server\windows-media
REM go to sub-dir(dirname is the mac address) of the samba share and run the auto script
for /F  "usebackq tokens=2 delims=:"  %%i IN (`ipconfig /all ^| find /i "physical address" `) DO cd %%i
REM -- end
$\>imagex /unmount c:\winpe_amd64\mount /commit
#####END script customization
$\>copy "C:\Program Files\Windows AIK\Tools\amd64\imagex.exe"  C:\winpe_amd64\ISO\
$\>copy C:\winpe_amd64\winpe.wim C:\winpe_amd64\ISO\sources\boot.wim
$\>Oscdimg -n -bC:\winpe_amd64\ C:\winpe_amd64\ISO C:\winpe_amd64\winpe_amd64.iso

How to do sysprep in Windows 2003?
On imagex/sysprep source computer:
Download the latest sysprep tool:
WindowsServer2003-KB926028-v2-x86-ENU.exe /x to extract the file
Extract files in to c:\sysprep
Run setupmgr.exe to create answer file c:\sysprep\Sysprep.inf
Run sysprep /reseal to prepare the OS for imaging
How to do sysprep in Windows 2008 R2?
On WAIK tool computer:
Open Windows system image manager?in WAIK to create unattended answer file
Click New answer file?to select a windows catalogue file (.clg) in installation media \Deploy\Operating Systems\Windows Server 2008 R2 x64\sources\.
In the Windows images?panel, right click various parameters to add to answer file
Detailed instruction:
On imagex/sysprep source computer:
Retrieve above new answer file *.xml, Run sysprep /generalize /oobe /shutdown /unattend:unattend.xml?
Capture image in imagex/sysprep source computer
This is manual process, because it is once-off job
Once the imagex/sysprep source computer is shutdown, Reboot it with winpe_amd64.iso by mounting to CDROM or loading via PXE
#The files source is the partition with windows folder, It could be d:, because W2k8 assign c: to the small system partition by default, which  doesn't need to be captured.
$\>imagex /compress fast  /capture [c:|d:]  $sambashare:\boot.wim "w2k8 R2 64bit"
Copy the boot.wim to samba share, which will be used by imagex /apply.
Apply image in imagex/sysprep target computer
This need to be full automatic process, boot winpe_amd64.iso via PXE. Once winPE is initialized, it will mount the sambashare, cd to its own directory and run auorun.bat, which will partition disk and apply the image.

It is zero touch installation, no human intervention is needed, however, unlike kickstart, computer name/IP address etc can't be configured automatically. Well, in theory, you can specify the information in sysprep then re-capture image for each unique configuration, but it defeats the purpose of saving time in provision. The other way is fresh installation by giving answer file in samba share to setup.exe.
 But it can only be used for Visa/2008/7/2008 R2, because there is no winPE which support running setup.exe of pre-Vista Windows

$ ls $sambashare\08-00-27-b2-40-3e
autorun.bat  boot.wim  diskpart.txt
$ cat $sambashare\08-00-27-b2-40-3e\autorun.bat 
diskpart /s diskpart.txt
imagex /apply boot.wim 1 c: && bcdboot c:\windows /s c: && exit
#bcdboot c:\windows is not needed for pre-Vista Windows, but it does no harm.

It is done, if you have setup answer file properly, the imagex/sysprep target computer should be able to boot smoothly without any human intervention.
Issues  encountered:
Q: DNS name is not resolvable in gPXE.
A: Make sure set use-cached 0 is set in pxelinux.gpxe, then recompile
$vi pxelinux.gpxe
set use-cached 0
dhcp net0
chain http://pxe-server/boot/home.php?mac=${net0/mac}

Q: gPXE couldn't boot Linux kernel with error:Kernel panic not syncing: VFS: unable to mount root fs
A:Make sure imgfree is the first statement in Linux pxe config
 $curl http://pxe-server/boot/home.php?mac=08:00:27:4d:4d:aa
kernel http://pxe-server/boot/os/centos-5.5-i386/pxeboot/vmlinuz ksdevice=link  ks=http://pxe-server/boot/hosts/linux-ks/08-00-27-4d-4d-aa.txt
initrd http://pxe-server/boot/os/centos-5.5-i386/pxeboot/initrd.img

Q:Error in imagex /apply after booting winPE via PXE: Not Enough Server Storage is available to process is command
A: Make sure the Windows VM have at least 1G RAM, because winPE ISO will consume a few hundreds MB as memdisk

Q:What if the winPE doesn't recognize the device (RAID/NIC) with default drivers.
A:Add customize driver to winPE, refer to

Thursday, October 13, 2011

Collectl, an all-in-one tool for collecting Linux statistical data

Collectl,collect for Linux, is a single tool which integrates functions of various tools:sar,iostat,mpstat,top,slaptop,netstat,nfstat,ps .. .
- Supported: Linux
- Requirement: Perl
Collectl features:
- run in command line or run as daemon
- Various output formats: raw,gunplot,gexprt(ganglia),sexpr,lexpr,csv(--sep ,)
- Send data to other programs (ganglia) remotely via socket instead of writing to a file
- IPMI monitoring for fans and temperature sensors
- Support module (Perl scripts)  for customized checks
- Monitor process’s disk read/write, find the top processes keeping disk busy
The last one is the most impressive feature, I haven’t found other Linux tools can do it. (DTtrace can in Solaris)
collectl  examples
#help, all options 
$collect –x
#-s?, what to monitor:c – cpu  d – disk “collectl   --showsubsys”
#-c 5 : collect 5 samples and exit
#-oT:  T - preface output with time only ; “collectl   --showoptions”
$collectl   -sc -c5 -i2 --verbose -oT
waiting for 2 second sample...
#Time      User  Nice   Sys  Wait   IRQ  Soft Steal  Idle  CPUs  Intr  Ctxsw  Proc  RunQ   Run   Avg1  Avg5 Avg15
12:39:34      0     0     0     0     0     1     0    97     1  1082     23     0    76     1   0.42  0.42  0.44
12:39:36      0     0     0     0     0     1     0    97     1  1088     24     0    76     1   0.42  0.42  0.44

The following demonstrates how collectl identify the process reading/writing most data to disk
#Hammer disk by writing 50mb data with dd
$dd if=/dev/urandom of=test bs=1k count=50000
#collectl identifies the “dd” process
#in top mode, sort by  “iokb   total I/O KB” ; “collectl –showtopopts”
$collectl -i2  --top iokb
TOP PROCESSES sorted by iokb (counters are /sec) 12:50:31
# PID  User     PR  PPID THRD S   VSZ   RSS CP  SysT  UsrT Pct  AccuTime  RKB  WKB MajF MinF Command
6861  root     18  6784    0 R    3M  572K  0  0.91  0.00  45   0:00.91    0 3680    0   97 dd
1  root     15     0    0 S    2M  632K  0  0.00  0.00   0   0:28.21    0    0    0    0 init
2  root     RT     1    0 S     0     0  0  0.00  0.00   0   0:00.00    0    0    0    0 migration/0

Tuesday, October 11, 2011

Understanding Red Hat Linux recovery runlevels

If Linux system can boot but hang during starting a service, booting to “recovery runlevels” can skip the service and gain shell to troubleshoot.
If Linux system can’t boot at all,  booting from rescue CD (first installation media) and type “linux rescue” to gain shell to troubleshoot
Red Hat Linux boot order
The BIOS ->MBR->Boot Loader->Kernel->/sbin/init->
/etc/rc.d/rcX.d/ #where X is run level in /etc/inittab
run script with K then script with S
Recovery runlevels
- runlevel  1
Execute up to /etc/rc.d/rc.sysinit and /etc/rc.d/rc1.d/
Runlevel 1 is identical to singleuser mode. It is switched to singleuser mode in last step, just a number of trivial scripts executed before that.
 $ls  /etc/rc.d/rc1.d/S*
 /etc/rc.d/rc1.d/S02lvm2-monitor  /etc/rc.d/rc1.d/S13cpuspeed  /etc/rc.d/rc1.d/S99singlesingleuser

- single
Execute up to /etc/rc.d/rc.sysinit

- Emergency
Does not execute /etc/rc.d/rc.sysinit.
 Because rc.sysinit is not executed, file system is mounted in read-only mode. You need run “mount –o rw,remount /” to remount it in read-write mode.
emergency runlevel is Red Hat term, it is identical to  “init=/bin/sh” in any Linux distribution
How to go to a  runlevel
In the grub menu, type “a” to append one of following options to boot line.
When Centos hung on starting up boot services, how to get to shell without rescue CD
RHCE Notes - Troubleshooting booting issue

Advanced RPM topics

“queryformat”  option can query every piece information of a rpm package, the  information tags (macros ) are returned  by “rpm –querytags” command
#list top 2 rpm packages sorted by installation time
$rpm -qa  | xargs -I{} rpm -q --queryformat "{}        %{installtime}\n" {} | sort -rn -k2 | head -2
collectl-3.5.1-1        1317864013
git-        1316484590
#unfortunately the time returned is unixtime.  You can convert it to human readable format by  “date –d @timestring” e.g 
$date -d @1317864013
Thu Oct  6 12:20:13 EST 2011
#but there is a shortcut  “--last”
$ rpm -qa --last  | head -2
collectl-3.5.1-1                              Thu 06 Oct 2011 12:20:13 PM EST
git-                             Tue 20 Sep 2011 12:09:50 PM EST

"rpm -qa" supports regular expression itself, rather than pipe to grep e.g “rpm -qa | grep perl”
“rpm –qa perl\*” also works. There is no improvement on speed but typing become lesser.
requires and provides
#You can check the package dependency before install the package
$ rpm -qp --requires git-1.7.6-1.el5.rf.i386.rpm
#To meet the dependency, you want to check who provides 
$yum whatprovides
openssl-0.9.8e-20.el5.i686 : The OpenSSL toolkit
Repo        : base
Matched from:
Other       :
#if openssl has been installed, “rpm -q –whatprovides” can also provide the answer
$rpm -q --whatprovides

rpm scriptlets

#query all nopre|nopost|nopreun|nopostun  scripts
$rpm -q --scripts xinetd
postinstall scriptlet (using /bin/sh):
if [ $1 = 1 ]; then
/sbin/chkconfig --add xinetd
preuninstall scriptlet (using /bin/sh):
if [ $1 = 0 ]; then
/sbin/service xinetd stop > /dev/null 2>&1
/sbin/chkconfig --del xinetd
postuninstall scriptlet (using /bin/sh):
if [ $1 -ge 1 ]; then
/sbin/service xinetd condrestart >/dev/null 2>&1
#query postinstall script only
$ rpm -q --queryformat "%{POSTIN}"  xinetd
if [ $1 = 1 ]; then
/sbin/chkconfig --add xinetd
#Don’t run the scripts during install/remove
rpm –i –noscripts|nopre|nopost|nopreun|nopostun   pkgname
rpm –e –noscripts|nopre|nopost|nopreun|nopostun   pkgname

Extract rpm contents without install

#use rpm2cpio to extract everything
$mkdir /tmp/epel
$ cd /tmp/epel
$ rpm2cpio /root/epel-release-5-4.noarch.rpm | cpio -ivd
#use rpm2cpio to extract particular file
$rpm2cpio /root/epel-release-5-4.noarch.rpm | cpio -ivd  ./usr/share/doc/epel-release-5
#another way is to use rpm install with alternative root
$ rpm --root /tmp/epel/ -ivh  --nodeps /root/epel-release-5-4.noarch.rpm
Recover corrupted rpm database
Build RPM from source file

Thursday, September 15, 2011

Get information for inode or block of a file and vice versa

Debugfs can retrieve a file’s inode or block information and vice versa, it is provided by e2fsprogs package, which should be installed by default.
Find a file’s information for inode or block
#find a file’s inode id by ls, the sample file’s inode number is 12
$ls -li /boot/message
12 -rw-r--r-- 1 root root 80032 Mar 13  2009 /boot/message
#find a file’s inode or block info by debugfs
$echo "stat message" | debugfs /dev/sda1
debugfs 1.39 (29-May-2006)
debugfs:  Inode: 12   Type: regular    Mode:  0644   Flags: 0x0   Generation: 3354433043
User:     0   Group:     0   Size: 80032
File ACL: 4640    Directory ACL: 0
Links: 1   Blockcount: 162
Fragment:  Address: 0    Number: 0    Size: 0
ctime: 0x4e2371f6 -- Mon Jul 18 09:36:22 2011
atime: 0x49b95d4d -- Fri Mar 13 06:06:53 2009
mtime: 0x49b95d4d -- Fri Mar 13 06:06:53 2009
(0-11):7681-7692, (IND):7693, (12-78):7694-7760
Find a file based on inode or block info
#find a file based on inode number by find, but it may take long time to search
$find /boot/ -inum 12
#find a file based on inode number by debugfs
$echo -e "ncheck 12" | debugfs /dev/sda1
debugfs:  Inode Pathname
12    /message
##find a file based on block number by debugfs
#use icheck to find inode number based on block number first
$echo -e "icheck 7693 " | debugfs /dev/sda1
debugfs:  Block Inode number
7693  12
#then use ncheck find the file by inode number
$echo -e "ncheck 12 " | debugfs /dev/sda1
debugfs:  Inode Pathname
12    /message

Recover corrupted RPM database

RPM database consists of a number Berkeley DB   files in  /var/lib/rpm, the exception  is  __db.* files, which like cache files are updated for every rpm operation and they can be safely deleted.
#tested in Centos 5.5 
$ ls /var/lib/rpm
Basenames     __db.001  __db.003  Filemd5s  Installtid  Packages      Provideversion  Requireversion  Sigmd5
Conflictname  __db.002  Dirnames  Group     Name         Providename   Pubkeys         Requirename  Sha1header      Triggername
$ file /var/lib/rpm/Packages
/var/lib/rpm/Packages: Berkeley DB (Hash, version 8, native byte-order)
If one of the DB files is partially corrupted and it is still readable by /usr/lib/rpm/rpmdb_dump, you can reload the DB file and rebuild db.
$cd /var/lib/rpm
$rm -f __db*      
$mv Packages Packages.orig
$/usr/lib/rpm/rpmdb_dump Packages.orig | /usr/lib/rpm/rpmdb_load Packages
$/usr/lib/rpm/rpmdb_verify Packages
#if you got this error: db_verify: PANIC: fatal region error detected; run recovery
#make sure /var/lib/rpm/__db.* are cleaned
#It is unlikely to rebuilddb if rpmdb_verify fails
$rpm -v –rebuilddb
If one of the DB files is completely corrupted and it is not readable by rpmdb_dump, you have to restore from backup,
$cd /var/lib/rpm
$cp Packages Packages.bak
#simulate a damaged RPM DB file
$ >Packages
$ cp Packages.bak  Packages
# Simply restoring  from backup file won’t work
#file verification is successful 
$ /usr/lib/rpm/rpmdb_verify Packages
#but any rpm operation fails
$rpm -qa 
error: rpmdbNextIterator: skipping h#     294 Header V3 DSA signature: BAD, key ID e8562897
#Even “rpm –rebuilddb” fails
$rm -f __db.*
$rpm –rebuilddb
error: rpmdbNextIterator: skipping h#     294 Header V3 DSA signature: BAD, key ID e8562897
#Notice the error about signature: BAD? The Pubkeys have to be cleaned as well.
$ mv Pubkeys Pubkeys.bak
#all good after removing Pubkeys file, a new Pubkeys is generated automatically on “rpm –rebuilddb”
$ rm -f __db.*
$ rpm –rebuilddb
$ rpm -qa | head -2

Thursday, August 4, 2011

Authenticate Linux users by Windows AD: LDAP+Kerberos or Winbind method

Authenticating Linux users by Windows AD has become popular in many organizations for the convenience of centralized account management. It makes sense, if you need existing Windows account to login to Linux servers, but it doesn’t make sense to install a new Windows AD for the sole purpose of authenticating Linux users, just because Windows has a nice GUI. Redhat Directory Server/Centos Directory Server/389 Directory Server all  are capable of doing such task beautifully by ldap+StartTLS or ldap+Kerberos.
At least, there are 4 ways to authenticate Linux users by Windows AD:
1.    Linux Clients use Linux LDAP server synchronized with Windows AD.
     Replicate a subtree of Windows AD accounts to Linux Ldap Server (Redhat Directory Server/Centos Directory Server/389 Directory Server), Linux users authenticate against the   Linux LDAP Server either by Ldap+StartTLS or Ldap+Kerberos
A more complex approach, it is for those who don’t like a Windows AD sitting in datacenter.
2.    Linux Client use LDAP or LDAP+SSL(ldaps) or LDAP+StartTLS
User information store: LDAP
Authentication: LDAP or LDAP+SSL(ldaps) or LDAP+StartTLS
Limitation: Client can login but can’t change its password
3.    Linux Client use ldap+Kerberos
User information store: LDAP
Authentication: Kerberos
The most popular approach, work out of the box, no need to install additional software. User can change  its password.
A vbscript to add unix user to windows AD.
4.    Linux Client use samba Winbind
User information store: Winbind local file or winbind ldap
Authentication: Winbind
Cons to solution #3 ldap+Kerberos approach:
- /etc/init.d/windbind dameon need to be running
- The Linux client  has to be joined to the domain, it is showed in computers container.
- The UID/GID is derived from Windows SID(ID = RID - BASE_RID + LOW RANGE ID, "man 8 idmap_rid"), you don’t seem to have a choice to set to a specific number.
Pros to solution #3 ldap+Kerberos approach:
- No need to insall Service for Unix in Windows
This post demonstrate method #3 ldap+kerberos and #4  Winbind.
Linux OS: Centos 5.5
Windows OS: Window 2003

Linux Clients use  ldap+Kerberos

Setup Windows Server
- Install  “Windows Services for UNIX Version 3.5” for Windows 2003
   Windows 2003 R2 or Windows 2008 has built-in component “Server for NIS Service”
- Create a generic user account for Linux nss-ldap client to run ldap query with the credentials.
     e.g username=Linux-bind-user  password=Redhat123
- Create a test group, go to “Unix Attribute tab” to set GID etc
- Create a test user, go to “Unix Attribute tab”, to set uid/homedir/shell/gid etc
- Modify default domain security group policy to allow users change password immediately, by default it is 1 day, change it to 0 day.
Set up Linux client
#make sure the following pkgs are installed 
#Edit configuration files: /etc/nsswitch.conf; /etc/ldap.conf; /etc/krb5.conf; /etc/pam.d/system-auth
#The easiest way is to run authconfig-tui.
#Before run the command, empty config files for easy reading
$cat /dev/null >/etc/ldap.conf ; cat /dev/null > /etc/krb5.conf
$authconfig-tui       #select use ldap for “User information” and “use md5 passwords, use shadow passwords, use Kerberos,local authorization is sufficient” for authentication.
$cat /etc/ldap.conf
uri ldap://
base cn=Users,dc=ad,dc=example,dc=com
ssl no
binddn cn=linux-bind-user,cn=users,dc=ad,dc=example,dc=com
bindpw Redhat123
##Map Unix account attributes to Winnows SFU 3.5 attributes
##NOTE: Windows 2003 R2/Windows 2008 use different name
##In either scheme, the attribute can be retrieved by ldapsearch command
#$ldapsearch –x –W –b ”cn=Users,dc=ad,dc=example,dc=com” –D “cn=linux-bind-user,cn=users,dc=ad,dc=example,dc=com” 
#msSFU30Name: jsmith
#msSFU30UidNumber: 10001
nss_map_objectclass posixAccount User
nss_map_objectclass shadowAccount User
nss_map_objectclass posixGroup Group
nss_map_attribute uid sAMAccountName
nss_map_attribute uidNumber msSFU30uidNumber
nss_map_attribute gidNumber msSFU30gidNumber
nss_map_attribute gecos         name
nss_map_attribute homeDirectory msSFU30HomeDirectory
nss_map_attribute loginShell msSFU30loginShell
pam_login_attribute sAMAccountName
pam_filter objectclass=User
nss_base_password cn=Users,dc=ad,dc=example,dc=com
nss_base_shadow cn=Users,dc=ad,dc=example,dc=com
nss_base_group cn=Users,dc=ad,dc=example,dc=com
pam_password ad
#Optional, "referrals yes" try to connect both ldap:// and ldap://
referrals       no
$cat /etc/krb5.conf
##[logging] section is not needed, I think, it is  for KDC daemon running locally only
default_realm = AD.EXAMPLE.COM                    #realm name MUST be in capitals 
dns_lookup_realm = false
dns_lookup_kdc = false
kdc =
admin_server =
##domain to realm mapping is also optional for user login, it is for the host added to KDC as principal to provide service
[domain_realm] = AD.EXAMPLE.COM = AD.EXAMPLE.COM

#nsswitch.conf system-auth don’t need to be further customization after “authconfig-tui” is run, just post them for reference
$cat /etc/nsswitch.conf
passwd:     files ldap
shadow:     files ldap
group:      files ldap
$grep krb5  /etc/pam.d/system-auth
auth        sufficient use_first_pass
account     [default=bad success=ok user_unknown=ignore]
password    sufficient use_authtok
session     optional

Test LDAP+Kerberos

#Query LDAP user information or run getent password|shadow|group to list all 
$id jsmith 
uid=10001(jsmith) gid=10000(unix-test-group) groups=10000(unix-test-group)

pam_krb5[5330]: password change failed for jsmith@AD.EXAMPLE.COM: Password change rejected, Password change rejected
test changing password in Windows GUI, Make sure the password meet password complexity policy and minimum change day is 0

 Authentication failure (Cannot find KDC for requested realm)
$ kinit
kinit(v5): Cannot find KDC for requested realm while getting initial credentials
check the relam spelling is correct and all in UPPER case, e.g AD.EXAMPLE.COM

 Authentication failure (Clock skew too great)
synchronize datetime with AD and enable ntpd
$ntpdate -s

Enable DEBUG:
$cat /etc/ldap.conf
 debug 7
Linux Clients use  samba Winbind

#Windows required software
Windows Services for UNIX is not needed
#Linux required software
#Clear /etc/ldap.conf and /etc/krb5.conf in last test
$cat /dev/null >/etc/ldap.conf; $cat /dev/null >/etc/krb5.conf; 
#select use winbind for “User information” and “use md5 passwords, use shadow passwords, use winbind,local authorization is sufficient” for authentication.
#supply following information and  type in "domain admin"'s username password to join domain
$cat /etc/samba/smb.conf
workgroup = AD
password server =
security = ads
idmap uid = 16777216-33554431
idmap gid = 16777216-33554431
template shell = /bin/bash
winbind offline logon = false
##above lines are generated by authconfig-tui
##following 3 parameters need mannual customization
winbind use default domain = true
winbind enum users = yes
winbind enum groups = yes
##Other files don't need further customization, just paste here for reference
$cat /etc/krb5.conf
default_realm = AD.EXAMPLE.COM
dns_lookup_realm = false
dns_lookup_kdc = false
kdc =
$grep winbind /etc/pam.d/system-auth
auth        sufficient use_first_pass
account     [default=bad success=ok user_unknown=ignore]
password    sufficient use_authtok
$grep winbind /etc/nsswitch.conf
passwd:     files winbind
shadow:     files winbind
group:      files winbind

Test winbind

$wbinfo -u
$gettent passwd
jsmith:*:16777216:16777216:John smith:/home/AD/jsmith:/bin/bash
##Group membership is added by opening the group, then click add member button in Windows GUI
$id jsmith
uid=16777216(jsmith) gid=16777216(domain users) groups=16777216(domain users),16777225(unix-test-group)

Sunday, July 31, 2011

Passed 3/5 RHCA: EX333 Security Network Services

I would probably rate EX333 as the most difficult exam among the 3 exams I have passed, Why  is it so difficult?

1. The exam objectives are related: for example, Kerberos depends on NIS, without a working NIS, you are doomed.

2. The exam is broken into morning section and afternoon section, You won’t pass the exam  if either section fails. but  you are still allowed to sit for the afternoon section despite  the result of morning section.

My blog post for EX333 study notes:

Authenticate BIND zone transfer with TSIG key

Setup Postfix SMTP password authentication with SASL

Wednesday, July 6, 2011

CentOS Directory Server 8 Quickstart

CentOS Directory Server  is a rebuild of the Red Hat Directory Server.  Red Hat Directory Server, Fedora 389 Directory server,  and Sun One Directory Server are similar, because they  all originated from Netscape Directory Server (NDS).  OpenLDAP is also a member of the extended family , whose root is University of Michigan slapd project, the parent of Netscape Directory Server.  The obvious difference is that OpenLDAP doesn't have built-in management console.
Install CentOS Directory Server (CDS)  8.1.0  on Centos  5.5
#CDS requires Centos 5.3 or newer
#install openldap-clients, as CDS ldap clients are not very friendly
$yum install java-1.6.0-openjdk openldap-clients centos-ds
#Link /usr/bin/java to the java 1.6 binary
$alternatives --config java

Setup  CentOS Directory Server

#Create ldap user/group for ldap daemon
$groupadd ldap; useradd -g ldap -s /sbin/nologin ldap
#Start installation wizard 
#start management console

CentOS Directory Server directory structure

/etc/init.d/dirsrv    #server startup script
/etc/init.d/dirsrv-admin   #admin server startup script
/etc/dirsrv/slapd-$instance/   #server config
/etc/dirsrv/slapd-$instance/dse.ldif   #server config for "cn=config"
/etc/dirsrv/slapd-$instance/scheme/99user.ldif #user defined scheme
/etc/dirsrv/admin-serv     #admin server config
/usr/lib/dirsrv/slapd-$instance/    #useful scripts: start&stop; backup&restore ...
/var/lib/dirsrv/slapd-$instance/db/      #database 
/var/lib/dirsrv/slapd-$instance/bak    #default backup dir
/var/log/dirsrv/slapd-$instance/    #logs
$ldapsearch -x -s base -b ""  # Root DSE; Show version, supported plugin etc

CentOS Directory Server backup and restore

#/etc/dirsrv/slapd-$instance/dse.ldif needs to be backup manually.
1) in GUI, select backup Directory Server
2) in CLI, /usr/lib/dirsrv/slapd-$instance/db2bak
#stop ldap server
$service dirsrv stop
#Restore using CLI, Usage: bak2db archivedir [-n backendname]
$/usr/lib/dirsrv/slapd-station08/bak2db /var/lib/dirsrv/slapd-station08/bak/station08-2011_06_30_15_11_51 -n userRoot
#By default,  backend instance name is  userRoot or NetscapeRoot
$grep  nsslapd-backend /etc/dirsrv/slapd-station08/dse.ldif
nsslapd-backend: userRoot
nsslapd-backend: NetscapeRoot

CentOS Directory Server export and import
##---------------------Export to ldif
1) in GUI, tasks -> export databases. 
2) in CLI, db2ldif
#Find out instance name
$ /usr/lib/dirsrv/slapd-$instance/suffix2instance -s "dc=stationn08, dc=example, dc=com"
Suffix, Instance name pair(s) under "dc=stationn08,dc=example,dc=com":
suffix "dc=station08, dc=example, dc=com"; instance name "userRoot"
$grep  nsslapd-backend /etc/dirsrv/slapd-station08/dse.ldif
nsslapd-backend: userRoot
nsslapd-backend: NetscapeRoot
#Export using backend instance name
$/usr/lib/dirsrv/slapd-$instance/db2ldif -n userRoot -a /tmp/all-userroot.ldif
#Export using suffix name
$/usr/lib/dirsrv/slapd-$instance/db2ldif -s  'dc=example,dc=com'  -a /tmp/all.ldif
##---------------------Import from ldif
#if Server is live
in GUI, Tasks->Import databases; 
in CLI, (It is recommended to use GUI for import due to the complexity of the script).
#If server is offline, use ldif2db script
$service dirsrv stop
$/usr/lib/dirsrv/slapd-$instance/ldif2db -n userRoot -i /tmp/all.ldif
LDAP command lines
CDS has built-in “mozldap-tools”, which have similar commands suites to openldap-clients, but Openldap-clients is easier to use because it support client configuration.
OpenLDAP client configuration files and command line options
##OpenLDAP client configuration files
$/etc/openldap/ldap.conf    #Global client conf, but BINDDN (Authenticated user)  is ignored  in Global conf
$HOME/ldaprc, $HOME/.ldaprc       #user ldap configuration file, set BINDDN here
$CWD/ldaprc                #local ldap configuration file
#Typical configuration
$cat /etc/openldap/ldap.conf
BASE    dc=station08, dc=example, dc=com
URI  ldap://
$cat /root/.ldaprc
BINDDN  cn=Directory Manager
##common command line options
-x         Simple authentication, not SASL bind
-W         prompt for bind password
-w passwd  bind password (for simple authentication)
-D binddn  bind DN    #username to authenticate
-b basedn  base dn for search 
-h host    LDAP server
-H URI     LDAP Uniform Resource Identifier(s)  #ldap://
-c         continuous operation mode (do not stop on errors), useful for skipping entries already exist when importing from ldif.
-Z         try to start TLS request (-ZZ to require successful response)  or -H ldaps://

Command line Search  Operation

# by default, search filter is  (objectclass=*) and display  ALL attributes. search is allowed for anonymous user, no password prompted
$ldapsearch -x
$ldapsearch -x -LLL    #less verbose, -LLL stripeout all comments
$ldapsearch -x  -s base    #(search scope). one of base, one(one-level sub), sub (whole subtree) or children,  default is sub
#Search filters, RFC 2254
#NO  ">" or" < "operator
= Exact match; >= greater than or equal; <= less than or equal; ~= aproximate match   
*  wildcard
#Logical operators
NOT   !   #( ! (uidNumber=500) )
OR   |   # (| (uidNumber>=502)(uid~=jim))
AND   &  # (& (uidNumber>=502)(uid~=jim))
#Escaped chars 
Character       ASCII value
*               0x2a
(               0x28
)               0x29
\               0x5c
NUL             0x00
Escaped "(" can be expressed as  "\(" or "\28"
e.g search telephoneNumber: (02)98660000 use ldapsearch -x  '(telephonenumber=\(02\)98660000)'
Command line  change operation 
Change operation need admin privilege which is specified by  BINDDN in /$HOME/.ldaprc or  "-D" in CLI

##types of changing whole record, 
changetype: add
changetype: delete
changetype: moddn
changetype: modify
##----------------------changetype: add
#default change type is add, no need to declare changetype: add,  as long as “-a” is specified in ldapmodify 
$cat /tmp/f1.ldif
dn: uid=jsmith,ou=People, dc=station08, dc=example, dc=com
uid: jsmith
givenName: john
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetorgperson
sn: smith
cn: john smith
$ldapmodify -x -W -a -f /tmp/f1.ldif
Enter LDAP Password:
adding new entry "uid=jsmith,ou=People, dc=station08, dc=example, dc=com"
##----------------------changetype: delete
#Option 1, use ldapmodify command
$cat /tmp/f1.ldif
dn: uid=jsmith,ou=People, dc=station08, dc=example, dc=com
changetype: delete
$ldapmodify -x -W -f /tmp/f1.ldif
Enter LDAP Password:
deleting entry "uid=jsmith,ou=People, dc=station08, dc=example, dc=com"
#Option 2, use ldapdelete command, note the ldif format difference
#ldapdelete support recursive delete with "-r"
$cat /tmp/f1.ldif
uid=jsmith,ou=People, dc=station08, dc=example, dc=com
$ldapdelete -x -W -f /tmp/f1.ldif
##----------------------changetype: moddn
#change RDN relative distingished name, the first part of DN
#Option 1, use ldapmodify command
$cat /tmp/f1.ldif
dn: uid=jsmith,ou=People, dc=station08, dc=example, dc=com
changetype: modrdn
newrdn: uid=jsmith2
deleteoldrdn: 1
$ldapmodify -x -W -f /tmp/f1.ldif
Enter LDAP Password:
modifying rdn of entry "uid=jsmith,ou=People, dc=station08, dc=example, dc=com"
rename completed
#Option 2, use ldapmodrn to achive same result
$ cat /tmp/f1.ldif
uid=jsmith,ou=People, dc=station08, dc=example, dc=com
$ldapmodrdn -x -r -W -f /tmp/f1.ldif
##----------------------changetype: modify
#add, replace,delete attributes of a record, not record itself
#multile actions separated by "-"
$cat /tmp/f1.ldif
dn: uid=jsmith,ou=People, dc=station08, dc=example, dc=com
changetype: modify
add: mail
delete: facsimileTelephoneNumber
replace: telephonenumber
telephonenumber: +1 408 555 1234
$ldapmodify -x -W -f /tmp/f1.ldif