Solutions Log by Dan Reiland

Spotlight Stops Indexing

Spotlight tips

This FAQ, derived from information in the "Spotlight" chapter of our book, Troubleshooting Mac OS X, Tiger Edition, addresses the following topics related to Spotlight™ in Mac® OS X 10.4 Tiger™.

* A brief tutorial on Spotlight.
* Rebuilding Spotlight indices.
* Stopping Spotlight indexing, which addresses:
o Anomalies from stopping indexing using Privacy.
o Disabling Spotlight indexing while indexing is in progress.
* Preventing Spotlight from indexing backup volumes.

A comprehensive treatment of Spotlight, including extensive troubleshooting information, can be found in the "Spotlight" chapter of our book, Troubleshooting Mac OS X, Tiger Edition.
A brief tutorial on Spotlight

* Spotlight will begin indexing any volume as soon as it is mounted, with certain exceptions: CDs, DVDs, and network-attached or shared volumes are not indexed when mounted. By volume we mean either a hard drive with a single partition, or a specific partition on a hard drive with multiple partitions.
* By default, Spotlight indexes and searches a limited number of folders, specifically those listed in the AppleCare® Knowledge Base document "Mac OS X 10.4: Where does Spotlight search?"
* Furthermore, Spotlight will neither index nor search:
o Hidden files: Files hose names begin with a period (.).
o Invisible files: Files whose invisible file-system attribute has been enabled, specifically the kIsInvisible Finder flag.
o Files within hidden or invisible folders.
o System-related files or folders.
* Spotlight saves all its index-related files in the .Spotlight-V100 directory at the root level of a volume it has indexed. The contents of a typical .Spotlight-V100 directory include the following files:
o .journalHistoryLog : Temporarily logs file system changes, such as new, updated, or deleted files. Spotlight then adjusts its indices accordingly. Changes noted in this file are performed so quickly that this file is normally 0 KB in size.
o .store.db : The journal file for the metadata index file store.db. This file is virtually a carbon copy of the store.db metadata index. It is updated first in case a crash occurs during metadata indexing.
o ContentIndex.db : The content index, used for finding files by their content.
o _IndexPolicy.plist : Indicates if indexing is enabled on the volume. The settings in this file are changed with the -i flag of the mdutil command.
o _exclusions.plist : Lists directories or folders which are prevented from being searched by Spotlight. The settings in this file are based on items listed in the Privacy tab of Spotlight preferences.
o _rules.plist : Lists directories on the volume to be excluded or included from indexing by Spotlight.
o store.db : The actual metadata index.
o mds-crash-state : This file only exists if mds, the main Spotlight process, crashes. This file’s presence in a volume’s .Spotlight-V100 directory:
+ Prevents Spotlight from crashing repeatedly.
+ Usually prevents Spotlight from updating the volume’s metadata store.
+ May indicate corrupted files on the volume.
* If you open Activity Monitor, the process mds is active when Spotlight is either indexing or searching. The mdimport process is also active during indexing.
* Spotlight employs SearchKit for content indexing and finding files by their content. SearchKit was first introduced in Mac OS X 10.3 Panther® for content indexing and Find By Content. However, Spotlight does not remove content indices created under previous versions of Mac OS X, which can consume considerable space on the hard drive.
* Removing these indices is especially tedious since Spotlight will not find hidden files whose names begin with a period (.) and index files created under Panther or Mac OS X 10.2 Jaguar have file names beginning with .FBC .
* As a result, you must use a utility like DEVONtechnologies EasyFind to locate and remove index files created under Panther or Jaguar.
* See our "Troubleshooting a failed indexing operation" FAQ for information about index files created by ContentIndexing under prior versions of Mac OS X.
* Tiger adds a variety of Terminal commands for working with Spotlight. These are:
o mdfind : Search Spotlight metadata indices.
o mdimport : Force Spotlight to import metadata for a specific directory.
o mdls : List the metadata attributes of a specific file.
o mdutil : A utility for managing Spotlight indices.
* You can find a variety of hints for manipulating the Spotlight index files, overcoming some of Spotlight's default limitations, and for using the Spotlight-specific Terminal commands by searching the Mac OS X Hints site for spotlight.
* Additional information about Spotlight can be found by reviewing the documents listed in the Related links section of this FAQ.

Rebuilding Spotlight indices

There are two approaches to rebuilding the Spotlight indices on a given volume:

* Using Privacy: Add, then remove, the volume whose Spotlight indices you wish to rebuild to the System Preferences > Spotlight > Privacy tab (hereafter abbreviated as "Privacy"). See the AppleCare Knowledge Base document "Spotlight: How to re-index files and folders."
* Using Terminal:

1.
Launch Terminal, which is in the Macintosh HD > Applications > Utilities folder.
2.

At the Terminal prompt, type the following command:

* sudo mdutil -E /path_to_volume

where /path_to_volume is the path of the volume whose Spotlight indices you wish to rebuild. [1]

For example, to rebuild the Spotlight indices for your Mac OS X startup disk, the command is:

* sudo mdutil -E /

3. Press Return. You will receive a confirmation message indicating that the index will be rebuilt automatically. Indexing will begin shortly thereafter.
4. At the Terminal prompt, type exit and press Return.
5. Quit (Command-Q) Terminal.
Stopping Spotlight indexing

A common practice to stop Spotlight™ from indexing volumes is to add the volume to the Privacy while the volume is being indexed.

Using Privacy to stop indexing can cause a variety of anomalous behaviors which are the result of inconsistencies in how the Privacy tab works vs. certain functions of the mdutil command in Tiger.
Anomalies from stopping indexing using Privacy

The following anomalous interactions between Privacy and mdutil, whether bugs or undocumented effects, were found testing with Tiger:

1. If you add a volume to Privacy while Spotlight is indexing that volume:
1. Spotlight immediately stops indexing the volume.
2. You will be unable to search that volume with Find (Finder > Command-F).
1. Find is implemented in Spotlight and respects the settings specified in the Privacy tab of Spotlight Preferences.
2. Adding a volume to Privacy creates the file _exclusions.plist in the volume's .Spotlight-V100 directory, with a setting specifying that the volume cannot be searched.
3. Incomplete index files may remain in the volume's .Spotlight-V100 directory.
4. The Spotlight indexing process may crash, leaving the file mds-crash-state in the volume's .Spotlight-V100 directory. This file appears to prevent Spotlight from indexing the volume again — even if the volume is removed from Privacy — until the volume's .Spotlight-V100 directory has been removed via the command:
1. sudo rm -ri /path_to_volume/.Spotlight-V100
5. where /path_to_volume is the path of the volume in question. [1]
6. The volume can then be reindexed by any of the following methods:
1. Restarting your Mac.
2. Using Disk Utility to first unmount, then mount the volume.
3. Issuing the command: sudo mdutil -E /path_to_volume
2. If you add a volume to Privacy after Spotlight has indexed that volume:
1. As noted previously, you will be unable to search the volume with Find.
2. Spotlight will no longer index the volume when its contents change. However, indexing is still enabled as can be shown by the command:
1. sudo mdutil -s /path_to_volume
3. The _exclusions.plist file, created when the volume was added to Privacy, prevents both searching the volume and subsequent indexing of that volume while it is listed in Privacy.
4. The ContentIndex.db file is removed from the volume's .Spotlight-V100 directory.
5. The metadata index file store.db in the volume's .Spotlight-V100 directory is reduced in size. Apparently, all metadata is removed from this file for files whose contents were included in the now-removed ContentIndex.db file.
6. Attempting to remove the remaining index files on the volume using the command:
1. sudo mdutil -E /path_to_volume
7. has no effect.
3. If you add a volume to Privacy and then disable indexing on that volume via the command:
1. sudo mdutil -i off /path_to_volume
2. The volume no longer appears in Privacy. If you attempt to add the volume to Privacy again, it will not appear in Privacy.
1. The volume will again appear in Privacy if you enable indexing with the command:
1. sudo mdutil -i on /path_to_volume
2. Despite this command, indexing is still disabled on the volume: the setting in the _exclusions.plist file in the volume's .Spotlight-V100 directory, created when it was added to Privacy, continues to prohibit indexing.
3. The general rule here is the following: you cannot add to Privacy any volume for which indexing has been disabled with the -i off flag of mdutil.
3. The volume can be searched to some degree with both Spotlight and Find.
1. Spotlight and Find ignore the setting in the _exclusions.plist file in the volume's .Spotlight-V100 directory, despite the fact that adding the volume to Privacy sets this file to indicate the volume cannot be searched.
2. The results of a search using Spotlight or Find for a string, such as xyz, on the volume will be equivalent to the results from the UNIX® find command:
1. find -x /path_to_volume -iname "*xyz*"
3. with the exception that neither Spotlight nor Find will not search within application packages whereas UNIX find will, the latter depending on permissions.
4. The general rule here is the following: disabling indexing on a volume with the -i off flag of mdutil after the volume has been added to Privacy enables the volume to be searched, to some degree, in both Spotlight and Find.
4. The only way to remove all the index-related files after stopping indexing via Privacy is to use mdutil to both disable indexing on the volume and erase the index files. This is accomplished by issuing the following two commands in Terminal in the order specified:
1. sudo mdutil -i off /path_to_volume
2. sudo mdutil -E /path_to_volume
5. While one may be tempted to combine both the -i off and -E flags in a single mdutil command, i.e.
1. sudo mdutil -i off -E /path_to_volume
6. we have seen mixed results from such: sometimes it works, sometimes it does not. Using the commands individually, in the order specified, works consistently.
7. You will then also be able to search the volume with Find or Spotlight as noted item 3 above. Once again, any volume added to Privacy will no longer appear in Privacy after indexing has been disabled on that volume via the -i off flag of mdutil.

Disabling Spotlight indexing while indexing is in progress

Adding a volume to Privacy while it is being indexed is not the best method for stopping the indexing of that volume. Instead, we recommend the following:
1. Let indexing begin on the volume.
2. Launch Terminal, which is in the Macintosh HD > Applications > Utilities folder.
3.

At the Terminal prompt, type the following command, exactly as written:

* sudo mdutil -i off /path_to_volume

where /path_to_volume is the path of the volume being indexed. [1]
4. Press Return.
5.

If prompted for a Password, type your Admin password, then press Return.

You will receive the response:

* /path_to_volume/: Indexing disabled for volume.

Spotlight will immediately cease to index the specified volume.
6.

Erase the partial index created by Spotlight. At the Terminal prompt, type the following command, exactly as written:

* sudo mdutil -E /path_to_volume

where /path_to_volume is the path of the volume being indexed. [1]
7. Press Return.
8.

If prompted for a Password, type your Admin password, then press Return.

You will receive the response:

* /path_to_volume/: Volume index removed.

9. At the Terminal prompt, type exit then press Return.
10. Quit (Command-Q) Terminal.
Preventing Spotlight from indexing backup volumes

If you use a backup or synchronization utility to duplicate one volume on your Mac — the source volume — to another volume, such as a FireWire drive — the target volume, you may notice that Spotlight will index the target volume even though you may have previously disabled indexing on such. This is especially true if:

* Your backup utility performs either:
o An incremental duplicate, copying only new or changed files from the source volume to the target volume.
o A full duplicate, copying all files from the source volume to the target volume, replacing the contents of the target volume.
* Spotlight indexing is enabled on the source volume.

As part of the duplication, your backup utility may copy the .Spotlight-V100 directory from the source volume to the target volume. If Spotlight is enabled on the source volume, then it will be enabled on the target volume after the .Spotlight-V100 directory is copied to such from the source volume.

As discussed earlier, the information Spotlight uses to ascertain if indexing is enabled on a volume is saved in the .Spotlight-V100 directory at the root level of that volume. Therefore, if you backup a volume on which indexing is permitted, after the /.Spotlight-V100 directory is copied from the source volume to the target volume, indexing will be enabled on the target volume.

Additionally, Spotlight will begin to index the target volume while the backup is in progress. This can slow the backup process considerably: in our tests, this has caused backups to require up to 150% of their normal time, i.e. a one-hour backup could require 1.5 hours due to Spotlight indexing the target volume.

To prevent Spotlight from indexing your backup volume both during and after a backup, we suggest the following:
1. Startup in Safe Mode. Spotlight is not launched in Safe Mode.

Note: When you startup in Safe Mode under Tiger:

* Only System fonts are available.
* Any fonts you disabled using Font Book will be enabled when you next restart or log in to your Mac.
* Certain Mac OS X features do not function, such as AirPort® cards.

For additional details, see:

* The "How disabled fonts are handled in Font Book 2.0 under Tiger" section of our "Undoing Font Book" FAQ.
* "What is Safe Boot, Safe Mode? (Mac OS X)."

2. Backup the source volume to the target volume using your backup utility.
3. Disable indexing on the target volume using Terminal:
3.1. Launch Terminal, which is in the Macintosh HD > Applications > Utilities folder.
3.2. At the Terminal prompt, type the following command, exactly as written:

* sudo defaults write /path_to_volume/.Spotlight-V100/_IndexPolicy Policy -int 3

where /path_to_volume is the path of the target volume for the backup performed in step 2. [1]

Note that this entire command should be typed as one continuous line without breaks. We use this command in lieu of mdutil as mdutil does not work in Safe Mode.
3.3. Press Return.
3.4. Type your Admin password when prompted, then press Return.
3.5. At the Terminal prompt, type exit then press Return.
3.6. Quit (Command-Q) Terminal.

When you next restart your computer normally, Spotlight will not index the target volume.
Related links

* All AppleCare Knowledge Base documents related to Spotlight and Tiger.
* Spotlight Technology Brief (PDF).
* Apple Developer Connection documents:
o Working with Spotlight.
o Spotlight Overview.
o Spotlight Query Programming Topics.
o Spotlight Importer Programming Guide.
o Spotlight Metadata Attributes Reference.
o Search Kit Programming Guide.
o Search Kit Reference.
* Finding corrupted files.

Notes

[1] The value for /path_to_volume in the Terminal commands specified in this FAQ depends on the volume in question:

* If the volume is your current Mac OS X startup disk, substitute the single character / for the strings /path_to_volume or /path_to_volume/ in the commands.
* If the volume is other than your current Mac OS X startup disk, substitute:
* /Volumes/volume_name
* for /path_to_volume, where volume_name is the name of the volume in question.
* Note that if volume_name contains spaces, either the entire path must be enclosed in double quotes or the spaces must be escaped with a reverse solidus, aka a backslash (\).
* Example: to stop and prohibit indexing on a volume named Backup Drive you would issue one of the following commands:
* sudo mdutil -i off /Volumes/Backup\ Drive
* sudo mdutil -i off "/Volumes/Backup Drive"
* The first escapes the space in Backup Drive with a backslash, while the second encloses the entire path in double quotes.

http://www.thexlab.com/faqs/stopspotlightindex.html#Anchor-Rebuilding-49575