Edit

Share via

  • Article

[Newsletters Archive ^] [< Volume 2, Number 1] [Volume 2, Number 3 >]

The Systems Internals Newsletter Volume 2, Number 2

http://www.sysinternals.com
Copyright © 2000 Mark Russinovich

March 27, 2000 - In this issue:

  1. EDITORIAL

    • Mark to coauthor "Inside Windows 2000, 3rd Ed." with David Solomon

  2. WHAT'S NEW AT SYSINTERNALS

    • PsKill v1.03, PsList v1.12
    • Junction v1.0
    • ElogList v1.0
    • NTFrob v1.6a
    • GetSid v1.1
    • HandleEx v2.23
    • Regmon v4.24, Filemon v4.28
    • AutoRuns v1.0
    • NT 4. ACL Editor
    • SysInternals T-Shirts
    • More SysInternals at microsoft.com
    • March/April Internals Columns
    • Not-So-New Stuff

  3. INTERNALS INFORMATION

    • Reparse points
    • The Kernel Handle Table
    • One or More Drivers Failed to Start
    • Microsoft NT-Related Patents

  4. WHAT'S COMING UP

    • TdiMon

SPONSOR: WINTERNALS SOFTWARE

The Systems Internals Newsletter is sponsored by Winternals Software, on the Web at http://www.winternals.com. Winternals Software is the leading developer and provider of advanced systems tools for Windows NT/2K. Winternals Software products include FAT32 for Windows NT 4.0, ERD Commander Professional Edition (advanced boot-disk capability for Windows NT), and Remote Recover.

Winternals Software's NTFSDOS Professional and NTFS for Win98 provide you full read and write access to your NTFS drives from DOS, Windows 95 and Windows 98. NTFSDOS Pro brings single-floppy "boot-disk" capability to Windows NT/2K. With NTFSDOS Pro you can delete buggy drivers, refresh files, and perform general file system maintenance on NTFS drives from a DOS boot floppy. NTFS for Win98 gives you transparent access to NTFS drives from Windows 95 and Windows 98. Easily share applications and files between NT and Win9x on NTFS drives in your dual-boot environment. Both utilities even have built-in NTFS Chkdsk capability. A free read-only version of NTFSDOS Pro is available at http://www.sysinternals.com/ntfspro.htm and a free read-only version of NTFS for Win98 is available at http://www.sysinternals.com/ntfs98.htm.

Hello everyone,

Welcome to the Systems Internals newsletter. The newsletter currently has 20,000 subscribers. The list's growth has been tremendous over the past two months with over 6,000 new subscribers! Please continue to pass the newsletter to your friends.

I'm pleased to announce that I'm coauthoring "Inside Windows 2000, 3rd Edition" with Dave Solomon (http://www.solsem.com). If you have an interest in Windows NT internals then you almost certainly have read Dave's "Inside Windows NT, 2nd Edition" (Microsoft Press). The release of Windows 2000 brings a number of changes to the NT kernel and surrounding components, some large and some small, and these changes mean that Dave's book requires a revision.

In many senses Dave and I have been independently working on "Inside Windows 2000" for the last three years, as we've kept up with the changes Microsoft has introduced to Windows 2000 in its evolution from NT 5.0 Beta 1 through Windows 2000 build 2195. I had actually been working on writing my own "Windows NT Internals" book for some time, but when the opportunity came to collaborate with Dave on the successor to a book of such high quality, and help write Microsoft's official view on Windows 2000 Internals, I couldn't pass it up.

Our decision to author the book together is a fairly recent one, so we've spent the past couple of months merging our research, notes, and articles. For totally unrelated reasons, we happen to live just twenty minutes from each other in a remote corner of Connecticut. Our proximity has enabled us to alternate days working at each other's houses, and we've recently spent many late nights researching and debating obscure details of the inner workings of Windows 2000. You can see a picture of Dave and I working at his house at http://www.sysinternals.com/inspic.jpg.

An unusual aspect of our partnership is that Dave has complete access to Windows 2000 source code whereas I do not (I've never had access to any Windows source code outside of that publicly available in the Device Driver Kit and the Installable File System Kit). Dave figures things out by walking through source code files, while I analyze listings produced by my custom disassembler and explore the guts of Windows 2000 on live systems with NuMega's SoftICE kernel-mode debugger. As a result, we each bring unique perspectives to the table, and have on more than one occasion jointly applied our respective resources to answer tough questions together.

Not only are we updating the original book to reflect Windows 2000 changes, we're also adding over 30% of brand-new content, including several new chapters. Topics we're introducing in the new book include booting, shutdown, crashes, storage management, service internals, Registry internals and WMI. This revision even comes with a CD-ROM containing a snap-shot of the SysInternals Web site, and a half-dozen tools I've written specifically for the book. One of the tools, LiveKd, lets you run the i386kd kernel debugger on a live system, which makes it easy to explore kernel internals without the hassle of serial cables and multiple computers. Without a doubt this book adds significant technical information and insight to the already solid foundation of its predecessor.

When will the book be available? Dave and I are heading out to Redmond the week of April 4 to get final technical review comments from Windows 2000 kernel developers, and Microsoft Press is saying that the book will be on shelves in July. I'll of course keep you posted in the newsletter.

Thanks!

-Mark

WHAT'S NEW AT SYSTEMS INTERNALS

PSKILL V1.03, PSLIST V1.12

PsList is a utility that lets you view detailed information about a local or remote system's active processes, and PsKill lets you kill processes on a local or remote system. These utilities have similar command-line syntax, where can you specify an optional computer name in the form "\\computer". If you include a user name with a computer name, the tools allow you to logon to the specified computer in a different user account than the one from which you are running the tools. These latest versions offer an alternate way for you to enter the password in environments where you are running them in front of other people and don't want to expose passwords. Now, if you include a computer account as an option but omit the password, they prompt you to enter the password and do not echo your input to the screen.

Download PsKill v1.03 at http://www.sysinternals.com/pskill.htm.
Download PsList v1.12 at http://www.sysinternals.com/pslist.htm.

JUNCTION V1.0

A form of symbolic link has finally come to Windows in the form of Windows 2000 NTFS junctions. Junctions are directory symbolic links, and the Windows 2000 Resource Kit includes a tool, linkd, that lets you create and delete junctions. Unfortunately, a basic Windows 2000 installation doesn't include any tools for creating junctions, and the Platform SDK documentation does not adequately document reparse points. These deficiencies lead me to implement Junction, a tool that not only allows you to create junctions, but to query files and display information on their reparse point if they have one. In order to aid developers wanting to implement their own reparse point tools, I've posted the full source code to Junction. See the section on reparse points later in the newsletter for more information on junctions and how Windows 2000 implements them.

Download Junction v1.0 with full source code at http://www.sysinternals.com/misc.htm.

ELOGLIST V1.0

The Windows 2000 Resource Kit include a tool named ELogDump that lets you dump records from an event log on the local or a remote computer. ELogList is an ElogDump clone that also lets you specify an optional account name and password so that you can access a computer's event logs from a different account than the one from which you are running the tool. ElogList is useful for dumping event logs from batch files, or capturing event log records into text files that you can import into spreadsheets for record keeping or analysis.

Download ElogList v1.0 at http://www.sysinternals.com/eloglist.htm.

NTFROB V1.6A

NTFrob is an applet that gives you more control over the foreground and background quantums (turns) that the Windows NT scheduler assigns to threads than what the Performance tab of the System control panel applet allows you. Using shorter quantums can improve the responsiveness of interactive applications, while longer quantums are more suited for long-running server workloads. NTFrob continues to keep pace with new service packs with its latest release, version 1.6a. Version 1.6a works on all released versions of NT 4.0 up through Service Pack 6a with the exception of Service Pack 6 (Microsoft withdrew Service Pack 6 shortly after its release because of a significant bug). A version of NTFrob that works on Win2K is coming soon.

Download NTFrob v1.6a at http://www.sysinternals.com/ntfrob.htm.

GETSID V1.1

If you are administering an environment where cloning eases rollout burdens then you'll probably be interested in GetSid. GetSid is like the Windows NT Resource Kit tool of the same name, but the SysInternals GetSid lets you obtain SIDs not just for user accounts, but for computers as well. Because GetSid works across the networks without requiring you to install any client software, you can easily use GetSid to verify that the computers on your network don't suffer from the duplicate SID problem that accompanies cloning.

Download GetSid v1.1 at http://www.sysinternals.com/misc.htm.
Learn about the duplicate SID problem at http://www.sysinternals.com/newsid.htm.

REGMON V4.24, FILEMON V4.28

In addition to displaying the names of processes performing file or Registry activity, these updates to Regmon and Filemon show you process identifiers as well. This enhancement helps you to distinguish file and Registry accesses between multiple processes having the same name.

Another enhancement present in these versions enables you to run Regmon and Filemon from remote Win2K terminal services sessions (as opposed to the console). The applications achieve this support because their GUIs check the operating system version number, and if running on Win2K, specify the "\\.\Global\" prefix to the name they use in their call to CreateFile when they open the device object of their driver component. In a terminal services environment the names device drivers assign to their objects are stored in the global (console) namespace, a namespace that is by default not visible in remote sessions. Remote sessions each have a local namespace. The "Global" prefix indicates to the Win2K object manager that the object manager should perform name lookups in the global namespace, rather than the namespace of the session where the lookup originates.

Download Regmon v4.24 at http://www.sysinternals.com/regmon.htm.
Download Filemon v4.28 at http://www.sysinternals.com/filemon.htm.

AUTORUNS V1.0

If you have a typical configuration, every time you boot your system and log in various components such as Explorer look in obscure Registry keys and folders and automatically run the programs referenced in them. David Solomon presents a list of all the locations where auto-run files are specified in his "Windows 2000 Internals" seminar (http://www.solsem.com), and Bryce Cogswell has taken the list and written AutoRuns, a program that lets you view their contents. You'll almost certainly be surprised at the hidden programs that run without your knowledge.

Download AutoRuns v1.0 at http://www.sysinternals.com/misc.htm.

HANDLEEX V2.23

The latest release of HandleEx adds a number of new user-interface usability enhancements, like the ability to ctrl-tab between its upper and lower views. More significantly, however, HandleEx now integrates with the Win2K security editor dialog interface, shows what memory mapped files a process has open, and displays the granted access mask for open handles.

When you switch HandleEx to its handle-based view it presents the files a process has opened via handles. In DLL-view HandleEx shows the files that a process has loaded as modules, and with the addition of memory-mapped file support, HandleEx lists the files a process has mapped via the Win32 memory-mapped file APIs. Since WinNT and Win2K do not let you delete files that processes have mapped, HandleEx's memory-mapped file support helps you determine what process is preventing you from deleting a file because of an outstanding mapping.

Download HandleEx v2.23 at http://www.sysinternals.com/handleex.htm.

NT 4 ACL EDITOR

Speaking of security editors, Microsoft documents the new Win2K security editor dialog APIs in the latest versions of the Platform SDK. However, the APIs of the NT 4 editors have always been undocumented and they remain that way. The NT 4 editors are the ones that you work with when you edit Registry key permissions in Regedt32 and NTFS file permissions in Explorer.

I determined the NT 4 security editor interfaces when I added object-security editing capability to our WinObj tool, and used the same interfaces to add security editing to HandleEx. I've finally decided to publish my documentation so that you can add native security editor functionality to the NT 4 versions of your own applications.

Get the NT 4 ACL Editor documentation at http://www.sysinternals.com/acledit.htm.
Download WinObj at http://www.sysinternals.com/winobj.htm.

SYSINTERNALS T-SHIRTS

If you like the technical information and utilities you get at SysInternals, show the world by wearing a SysInternals t-shirt. The t-shirts are 100% cotton Hanes Beef-T's, are printed in striking colors on both front and back, and cost only $14.95. In addition, $5 of every sale goes to the American Cancer Society.

View and order SysInternals t-shirts at http://www.sysinternals.com/tshirt.htm.

MORE SYSINTERNALS AT WWW.MICROSOFT.COM

I'm proud to say that the number of Microsoft Knowledge Base articles that refer users to SysInternals tools continues to grow. Here is the list of the recent additions I've tracked down.

  • Q243583 PRB: Mib.bin Causes Visual Studio Setup to Fail http://support.microsoft.com/support/kb/articles/Q243/5/83.ASP
    This article recommends the use of Filemon to track down Visual Studio setup errors.

  • Q242131 HOWTO: Display a List of Processes that Have Files Open http://support.microsoft.com/support/kb/articles/Q242/1/31.ASP
    Microsoft points users to HandleEx as a utility that shows what files processes have open.

  • Q232060 HOWTO: MDAC Setup Troubleshooting Guide http://support.microsoft.com/support/kb/articles/q232/0/60.asp
    DLLView and HandleEx get the spotlight in this article, which instructs users to use the tools to locate processes having Microsoft Data Access Component DLLs so that the user can terminate them before reinstalling MDAC.

  • Q245068 ERRMSG: Access is Denied. You Don't Have Permissions or the File is in Use http://support.microsoft.com/support/kb/articles/q245/0/68.asp?lng=eng&sa=allkb
    NtHandle gets the reference again in this article, which tells you how to determine which process has a file in use when you get an error deleting it.

  • Q247957 SAMPLE: Using DUPS.EXE to Resolve DLL Compatibility Problems http://support.microsoft.com/support/kb/articles/q247/9/57.asp
    This article references ListDLLs, DllView, and HandleEx as tools that enable you to track down DLL version problems.

Not only has SysInternals been featured in all these new KB articles, Rick Anderson, the author of the DUPS utility presented in the last KB article, references ListDLLs in his MSDN News article, "The End of DLL Hell". You can view the article on-line at Microsoft here: http://www.msdn.microsoft.com/isapi/msdnlib.idc?theURL=/library/techart/DLLdanger1.htm. The attention ListDLLs received as a result of this prompted me to port it to Windows 9x, so version 2.21 works on Windows 9x, Windows NT, and Windows 2000.

Download ListDLLs v2.21 at http://www.sysinternals.com/listdlls.htm.

MARCH/APRIL INTERNALS COLUMNS

Check out the March and April issues of Windows 2000 Magazine for my two-part "Internals" column on Windows NT and Windows 2000 storage management. In the first part of the series I describe NT 4 disk partitioning, advanced volume configuration, drive-letter assignment and device driver storage architecture.

In Part 2 I cover the storage changes present in Win2K, including new storage-management device drivers, dynamic-disk partitioning, support for advanced volumes that do not require reboots for reconfiguration, and Win2K's drive-letter assignment mechanisms.

INTERNALS INFORMATION

REPARSE POINTS

For some reason people always get excited about file system tricks, and Windows 2000 includes several new ones. Prior to Windows 2000, all Microsoft file systems have lack a feature familiar to UNIX users: the symbolic link. Symbolic links let you create a file or directory that references another file or directory somewhere else in the file system namespace. When an application accesses a link it actually accesses the target of the link. For example, if the link C:\drivers refers to the directory C:\winnt\system32\drivers, then a reference to the file name C:\drivers\ntfs.sys resolves to C:\winnt\system32\drivers\ntfs.sys.

NTFS version 5, the NTFS revision included with Windows 2000, supports a mechanism called reparse points. A reparse point is a block of data associated with a file or directory that contains a "tag" and information defined by the driver responsible for managing reparse points having that tag. Microsoft defines several built-in tags, including the junction and mount point tags. When NTFS encounters a reparse point while looking up a file name it aborts the lookup and returns a STATUS_REPARSE code to its caller. File system filter drivers and the I/O Manager watch for reparse codes that to their tag, and react in one of several ways. Hierarchical Storage Management (HSM) reparse points denote files that the HSM subsystem has moved to remote storage (such as tape), the Remote Storage filter driver (RsFilter.sys), for example, transparently pulls the file's data off of remote storage, removes the reparse point, and lets the file lookup retry.

A filter driver can also change the name of the file being opened. Mount point tags represent volume mount points, and allow you to connect volumes together from within their name spaces. Thus, you could mount a volume containing your project documents on the \projects directory of your C: drive. Doing this organizes your file system data and enables you to avoid using DOS-style drive letters.

Junctions resemble mount points, but instead of linking a directory to a volume, they link directories to other directories. They are NTFS' symbolic link support. At this point you're probably wondering why Microsoft didn't include file-based symbolic links. The answer is that symbolic links would wreak havoc with existing Win32 applications.

While there are many typical application behaviors that would cause unexpected results when working with files that are really symbolic links, deleting a symbolic link with a link-unaware program is a simple example. Consider a file stored in a central location that several symbolic links refer to. A user deleting one of the links probably intends to just delete the link and not the file itself. However, if the program is unaware of symbolic links it won't detect that the file is really a link and prompt you for the behavior you want. The problems become more severe when an application creates files that are related to the one they reference through a link should the related files be stored in the directory where the link target is located or in the directory where the symbolic link resides?

Microsoft was faced with a difficult problem, and I'm sure that there are applications that perform even more complex file manipulation that would break even if you could find a work-around for the issues I've mentioned. For this reason, I think it's unlikely that we'll ever see file-based symbolic links in Windows.

THE KERNEL HANDLE TABLE

The Win2K object manager introduced a new type of handle table that improves the performance of certain types of device drivers. Some drivers need to open handles to system objects while they are running in a user process' security context. Device drivers can bypass security checks when opening objects, so they must take steps to prevent a security hole that results when they create a handle to a sensitive object in the handle table of an unprivileged process. NT 4 drivers avoiding security problems have to either queue work items to worker threads that run in the system process context, or use the KeAttachProcess API to switch to the System process' handle table and address space. Both of these alternatives can degrade performance, especially if the driver must perform them frequently.

Win2K's solution to this problem comes in the form of a new flag that a driver can pass in an OBJECT_ATTRIBUTES structure when they are opening an object and obtaining a handle. The flag is not documented in the DDK, but is defined in the NTDEF.H header file as OBJ_KERNEL_HANDLE. When the object manager has opened the specified object and is creating a handle to return to the caller, it checks to see if this flag is present. If so, it creates the handle in a handle table named `ObpKernelHandleTable instead of the handle table of the currently executing process. The handles the object manager returns to callers requesting kernel handles have their high bit set, so all kernel handles have values greater than 0x80000000.

Whenever the object manager is passed a handle, in a call to ObReferenceObjectByHandle for example, where it must translate a handle to a pointer to the handle's underlying object, it checks to see if the handle reference is a kernel-mode reference and if the high bit of the handle is set. For references that match these criteria the object manager looks up the handle in the kernel handle table instead of the handle table of the currently executing process. Thus, a driver making references to kernel handles for secure objects avoids taking a performance hit and causing opening security holes.

ONE OR MORE DRIVERS FAIL TO START

Here's a piece of interesting Windows NT/2K trivia for you. When you see a dialog box during the boot that tells you that "One or more drivers failed to start", it's the Service Control Manager (SCM) that is both making the determination and presenting the dialog.

How does the SCM know that a driver failed to start? The SCM scans the HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services Registry key when it initializes, looking for device drivers that have Start values that specify that they start as boot or system-start drivers. When it finds an entry it opens the object manager namespace and looks to see if the device driver's name is present in the \Drivers directory. When a device driver starts successfully the I/O Manager places its driver object in the \Drivers directory, so when the SCM can't find the name it assumes that the driver didn't start successfully. You can view the contents of the \Drivers directory using WinObj.

Download WinObj at http://www.sysinternals.com/winobj.htm.

Software patents have gotten a lot of attention lately. Its seems that the US Patent and Trademark Office is granting patents to even the most obvious "innovations", and companies are taking advantage of the liberal patenting policy to lock down key technology for their exclusive use. Microsoft is no stranger to the patent game, but what many people don't know is that Microsoft owns patents to several key ideas that they developed with Windows NT. Here are a list of the kernel-related patents I've uncovered at IBM's Patent Server Web site:

"System for performing asynchronous file operations requested by runnable threads by processing completion messages with different queue thread and checking for completion by runnable threads"
http://www.patents.ibm.com/details?pn=US05758184__

This patent covers the fundamental ideas behind the advanced NT synchronization mechanism called completion ports. Completion ports allow processes to efficiently wait for I/O on many different object, and use support in the Windows NT/2000 scheduler to allow threads associated with a completion port to effectively utilize a multiprocessor.
Read more about completion ports at http://www.sysinternals.com/comport.htm.

"Server impersonation of client processes in an object based computer operating system"
http://www.patents.ibm.com/details?&pn10=US05187790

Impersonation is a powerful feature of the Windows NT/2000 security model that allows a server thread to temporarily adopt the security context of a client thread when the server is performing activities on behalf of the client. This allows the server to easily take advantage of the Windows NT/2000 security model when it accesses protected objects for a client. It's a clever, if not somewhat obvious, approach to distributed security and Microsoft owns the patent on it.

"Waitable object creation system and method in an object based computer operating system"
http://www.patents.ibm.com/details?&pn10=US05057996

"Conditional object creating system having different object pointers for accessing a set of data structure objects"
http://www.patents.ibm.com/details?&pn10=US05129083

"Object container transfer system and method in an object based computer operating system"
http://www.patents.ibm.com/details?&pn10=US05129084

"Temporary object handling system and method in an object based computer operating system"
http://www.patents.ibm.com/details?&pn10=US05136712

"Object transferring system and method in an object based computer operating system"
http://www.patents.ibm.com/details?&pn10=US05297283

The Windows NT/2000 object manager implements a namespace not unlike the Virtual File System (VFS) namespace present on UNIX implementations. Some of the patents Microsoft has obtained on the object manager really look like patents on object-oriented design, and several look like they overlap. However, I'm not a patent attorney, so that may just be my unenlightened view point.

If you look at the patents you'll find the names of several of the core Windows NT kernel developers, including David Cutler. Interestingly, some of the patents were obtained for Digital Equipment Corp of Maynard, MA by NT developers when they worked on VMS, but the patent process took so long that the patents list them as residing in Redmond, WA.

WHAT'S COMING UP

TDIMON

Have you ever wanted watch the TCP and UDP network activity in real-time, and know what exactly which processes are performing the activity? Stay tuned for TdiMon, a powerful addition to the SysInternals monitoring toolkit.

Thank you for reading the Systems Internals Newsletter.

Published Monday, March 27, 2000 7:09 PM by ottoh

[Newsletters Archive ^] [< Volume 2, Number 1] [Volume 2, Number 3 >]