Home » Public Forums » FoxTrot Search User Forum » Indexing Apple Email runs very slowly
| Indexing Apple Email runs very slowly [message #477] |
Thu, 16 February 2017 20:16  |
Robert White
Messages: 6
Registered: February 2017
|
Junior Member |
|
|
I set up an index for Apple Mail using the predefined location. I have it
set to index content. It starts indexing OK (e.g. it moves along pretty
quickly) but then it starts to run slower and slower and eventually appears
to freeze. I let it run for two days and it still had not completed. I am
not sure how many email messages there are - probably close to 100,000. I
am not sure if there is something the User\Library\Mail folder that is
hanging it up. Any advice? Thanks
|
|
|
|
| Re: Indexing Apple Email runs very slowly [message #479 is a reply to message #477] |
Fri, 17 February 2017 20:12  |
FoxTrot Engineering
Messages: 427
Registered: April 2020
|
Senior Member |
|
|
Robert White wrote:
> I set up an index for Apple Mail using the predefined location. I have it
> set to index content. It starts indexing OK (e.g. it moves along pretty
> quickly) but then it starts to run slower and slower and eventually appears
> to freeze. I let it run for two days and it still had not completed. I am
> not sure how many email messages there are - probably close to 100,000. I
> am not sure if there is something the User\Library\Mail folder that is
> hanging it up. Any advice? Thanks
You can open the indexer and crawler logs ("configuration" pane of the "manage indices" window, and check if anything interesting is being logged.
You can open Activity Monitor, check if FoxTrot Indexer and/or FoxTrot Crawler is using a lot of CPU, then select these processes in the list, and "sample process"; send me the result file directly.
You can also check "resource hogs" (at the bottom of the indexed locations, in the "indexed data" pane); in case some messages with large attachments are indexed as raw text (because of a problem in Mail's importer, or because of malformed messages), this could cause some performance problems in FoxTrot.
Jérôme - CTM Engineering
------------------------------------------------------------ ---------
"With FoxTrot I find files I do not find with Spotlight.
With FoxTrot I do not get swamped with 2000 found files in one bunch,
under "Documents" and now I have to scroll through a long list - with
FoxTrot I can quickly and easy narrow my search.
With FoxTrot with one click I can see a preview with the search term
highlighted.
One of the very important reasons I like FoxTrot is that it's results
seem so much more appropriate.
And searches in FoxTrot returns are almost instantly - way way faster
than in Spotlight"
Marlyse Comte, FoxTrot Personal Search user
Download a demo version from www.foxtrot.ch
------------------------------------------------------------ ---------
Jérôme - FoxTrot Engineering
|
|
|
|
| Re: Indexing Apple Email runs very slowly [message #480 is a reply to message #479] |
Sat, 18 February 2017 05:57 |
Robert White
Messages: 6
Registered: February 2017
|
Junior Member |
|
|
When I open the crawler logs it appears to be indexing various emails
(shows a long list of email type information). But the Indexer Log seems
to show that it keeps timing out. I have copied the last few lines below.
2017/02/17, 19:12:05 -0800: ### indexer busy; extending crawler timeout by
198 seconds ###
2017/02/17, 19:23:40 -0800: ### indexer busy; extending crawler timeout by
694 seconds ###
2017/02/17, 19:35:58 -0800: ### indexer busy; extending crawler timeout by
738 seconds ###
2017/02/17, 20:05:16 -0800: ### indexer busy; extending crawler timeout by
1758 seconds ###
2017/02/17, 20:30:36 -0800: ### indexer busy; extending crawler timeout by
1520 seconds ###
The index was created from a "predefined location" (Mail Icon) and in the
Manage Indexes panel does not allow you to go to an "indexed data" panel
and thus does not allow you to check for resource hogs. For reference, for
an index I created with some custom locations, the indexed data panel opens
and you can check for resource hogs.
In the Activity Monitor the CPU usage shows that FTProFileIndexer is using
consistently about 95% of CPU resources and when you click on the Disk tab
under the column Bytes Written it is up around 434.4 gigabytes.
I have attached two sample files from Activity Monitor. One is
FTProFileIndexer and the other is FoxTrot Indexer (Mail).
Hope all this helps pinpoint the problem. The indexing of regular folders
(e.g. docs, PDF and so forth) works fine. Only the mail indexer has this
problem.
On Friday, February 17, 2017 at 11:12:41 AM UTC-8, FoxTrot Engineering
wrote:
>
> Robert White wrote:
>
>> I set up an index for Apple Mail using the predefined location. I have
> it
>> set to index content. It starts indexing OK (e.g. it moves along pretty
>> quickly) but then it starts to run slower and slower and eventually
> appears
>> to freeze. I let it run for two days and it still had not completed. I
> am
>> not sure how many email messages there are - probably close to 100,000.
> I
>> am not sure if there is something the User\Library\Mail folder that is
>> hanging it up. Any advice? Thanks
>
> You can open the indexer and crawler logs ("configuration" pane of the
> "manage indices" window, and check if anything interesting is being logged.
> You can open Activity Monitor, check if FoxTrot Indexer and/or FoxTrot
> Crawler is using a lot of CPU, then select these processes in the list, and
> "sample process"; send me the result file directly.
> You can also check "resource hogs" (at the bottom of the indexed
> locations, in the "indexed data" pane); in case some messages with large
> attachments are indexed as raw text (because of a problem in Mail's
> importer, or because of malformed messages), this could cause some
> performance problems in FoxTrot.
>
>
> Jérôme - CTM Engineering
>
>
> "With FoxTrot I find files I do not find with Spotlight.
> With FoxTrot I do not get swamped with 2000 found files in one bunch,
> under "Documents" and now I have to scroll through a long list - with
> FoxTrot I can quickly and easy narrow my search.
> With FoxTrot with one click I can see a preview with the search term
> highlighted.
> One of the very important reasons I like FoxTrot is that it's results
> seem so much more appropriate.
> And searches in FoxTrot returns are almost instantly - way way faster
> than in Spotlight"
> Marlyse Comte, FoxTrot Personal Search user
>
> Download a demo version from www.foxtrot.ch
>
>
>
>
|
|
|
|
| Re: Indexing Apple Email runs very slowly [message #481 is a reply to message #479] |
Sat, 18 February 2017 06:20 |
Robert White
Messages: 6
Registered: February 2017
|
Junior Member |
|
|
One additional thing. I stopped indexing the two indexes I have. I quit
Fox Trot. And when I opened activity monitor it showed under CPU that
there were still two Fox Trot processes running.
On Friday, February 17, 2017 at 11:12:41 AM UTC-8, FoxTrot Engineering
wrote:
>
> Robert White wrote:
>
>> I set up an index for Apple Mail using the predefined location. I have
> it
>> set to index content. It starts indexing OK (e.g. it moves along pretty
>> quickly) but then it starts to run slower and slower and eventually
> appears
>> to freeze. I let it run for two days and it still had not completed. I
> am
>> not sure how many email messages there are - probably close to 100,000.
> I
>> am not sure if there is something the User\Library\Mail folder that is
>> hanging it up. Any advice? Thanks
>
> You can open the indexer and crawler logs ("configuration" pane of the
> "manage indices" window, and check if anything interesting is being logged.
> You can open Activity Monitor, check if FoxTrot Indexer and/or FoxTrot
> Crawler is using a lot of CPU, then select these processes in the list, and
> "sample process"; send me the result file directly.
> You can also check "resource hogs" (at the bottom of the indexed
> locations, in the "indexed data" pane); in case some messages with large
> attachments are indexed as raw text (because of a problem in Mail's
> importer, or because of malformed messages), this could cause some
> performance problems in FoxTrot.
>
>
> Jérôme - CTM Engineering
>
>
> "With FoxTrot I find files I do not find with Spotlight.
> With FoxTrot I do not get swamped with 2000 found files in one bunch,
> under "Documents" and now I have to scroll through a long list - with
> FoxTrot I can quickly and easy narrow my search.
> With FoxTrot with one click I can see a preview with the search term
> highlighted.
> One of the very important reasons I like FoxTrot is that it's results
> seem so much more appropriate.
> And searches in FoxTrot returns are almost instantly - way way faster
> than in Spotlight"
> Marlyse Comte, FoxTrot Personal Search user
>
> Download a demo version from www.foxtrot.ch
>
>
>
>
|
|
|
|
| Re: Indexing Apple Email runs very slowly [message #482 is a reply to message #480] |
Sat, 18 February 2017 10:33 |
FoxTrot Engineering
Messages: 427
Registered: April 2020
|
Senior Member |
|
|
Robert White wrote:
> The index was created from a "predefined location" (Mail Icon) and in the
> Manage Indexes panel does not allow you to go to an "indexed data" panel
> and thus does not allow you to check for resource hogs.
The "Resource Hogs" button is at the bottom of the "Predefined Locations" list where you selected "Mail".
> 2017/02/17, 19:12:05 -0800: ### indexer busy; extending crawler timeout by
> 198 seconds ###
> I have attached two sample files from Activity Monitor. One is
> FTProFileIndexer and the other is FoxTrot Indexer (Mail).
I would think that some large binary data (attachments) are being indexed as plain text. What is your macOS version? Can you copy and paste the following command in a Terminal window, then paste back the result?
mdimport -L
> One additional thing. I stopped indexing the two indexes I have. I quit
> Fox Trot. And when I opened activity monitor it showed under CPU that
> there were still two Fox Trot processes running.
Either reboot your Mac, or press the command and option keys while launching FoxTrot; then check the "Force-quit agents" checkbox.
Jérôme - CTM Engineering
------------------------------------------------------------ ---------
"FoxTrot is way better and faster than Spotlight and seriously more
usable. With around 200000 docs on my 60Gig drive, using Spotlight
is just hopeless.
Searches with FoxTrot are just instantaneous and you can view them in
all sorts of ways. The preview is just wonderful."
FoxTrot Personal Search user comment on www.versiontracker.com
Download a demo version from www.foxtrot.ch
------------------------------------------------------------ ---------
Jérôme - FoxTrot Engineering
|
|
|
|
| Re: Indexing Apple Email runs very slowly [message #483 is a reply to message #477] |
Sat, 18 February 2017 20:46 |
Robert White
Messages: 6
Registered: February 2017
|
Junior Member |
|
|
This is from Terminal
Last login: Sat Feb 18 11:42:27 on ttys000
Roberts-MacBook-Pro:~ RMW$
Roberts-MacBook-Pro:~ RMW$ mdimport -L
2017-02-18 11:43:19.558 mdimport[2679:162580] Paths: id(501) (
"/Library/Spotlight/iBooksAuthor.mdimporter",
"/Library/Spotlight/iWork.mdimporter",
"/Library/Spotlight/Microsoft Office.mdimporter",
"/System/Library/Spotlight/Application.mdimporter",
"/System/Library/Spotlight/Archives.mdimporter",
"/System/Library/Spotlight/Audio.mdimporter",
"/System/Library/Spotlight/Automator.mdimporter",
"/System/Library/Spotlight/Bookmarks.mdimporter",
"/System/Library/Spotlight/Chat.mdimporter",
"/System/Library/Spotlight/CoreMedia.mdimporter",
"/System/Library/Spotlight/Font.mdimporter",
"/System/Library/Spotlight/iCal.mdimporter",
"/System/Library/Spotlight/Image.mdimporter",
"/System/Library/Spotlight/iPhoto.mdimporter",
"/System/Library/Spotlight/iPhoto8.mdimporter",
"/System/Library/Spotlight/Mail.mdimporter",
"/System/Library/Spotlight/MIDI.mdimporter",
"/System/Library/Spotlight/Notes.mdimporter",
"/System/Library/Spotlight/PDF.mdimporter",
"/System/Library/Spotlight/PS.mdimporter",
"/System/Library/Spotlight/QuartzComposer.mdimporter",
"/System/Library/Spotlight/RichText.mdimporter",
"/System/Library/Spotlight/SystemPrefs.mdimporter",
"/System/Library/Spotlight/vCard.mdimporter",
"/Applications/Google Earth.app/Contents/Library/Spotlight/Google
Earth.mdimporter",
" /Applications/NeoOffice.app/Contents/Library/Spotlight/neoli ght.mdimporter ",
"/Applications/Xcode.app/Contents/Applications/Application
Loader.app/Contents/Library/Spotlight/MZSpotlight.mdimporter ",
" /Applications/Pixelmator.app/Contents/Library/Spotlight/Pixe lmator.mdimporter ",
" /Applications/Xcode.app/Contents/Library/Spotlight/uuid.mdim porter ",
" /Applications/GarageBand.app/Contents/Library/Spotlight/Gara geBandSpotlightImporter.mdimporter ",
" /Applications/GarageBand.app/Contents/Library/Spotlight/Logi cX_MDImport.mdimporter ",
"/Applications/Microsoft
Outlook.app/Contents/Library/Spotlight/Microsoft Outlook Spotlight
Importer.mdimporter"
)
Roberts-MacBook-Pro:~ RMW$
Also, when I first opened Fox Trot, I was able to see the Indexed Files tab
for the Mail index. I went to it and there were 100 files under Resources
Hog so I blacklisted them. After going out of Manage Indexes and back in,
for the Mail index there are only two tabs - Configuration and Sharing.
This has happened before - for the predefined index some of the tabs just
don't appear. Why the Indexed Files tab showed up at first and not now I
don't understand.
On Thursday, February 16, 2017 at 11:16:26 AM UTC-8, Robert White wrote:
>
> I set up an index for Apple Mail using the predefined location. I have it
> set to index content. It starts indexing OK (e.g. it moves along pretty
> quickly) but then it starts to run slower and slower and eventually appears
> to freeze. I let it run for two days and it still had not completed. I am
> not sure how many email messages there are - probably close to 100,000. I
> am not sure if there is something the User\Library\Mail folder that is
> hanging it up. Any advice? Thanks
>
|
|
|
|
| Re: Indexing Apple Email runs very slowly [message #484 is a reply to message #483] |
Mon, 20 February 2017 12:00 |
FoxTrot Engineering
Messages: 427
Registered: April 2020
|
Senior Member |
|
|
Robert White wrote:
> After going out of Manage Indexes and back in,
> for the Mail index there are only two tabs - Configuration and Sharing.
> This has happened before - for the predefined index some of the tabs just
> don't appear. Why the Indexed Files tab showed up at first and not now I
> don't understand.
This happens when the indexer is busy, and can't answer to the application's administrative requests. The only thing you can do in this situation is click "Stop" to stop the indexer, which should eventually happen after a moment. However, when you start it again, it should resume indexing and may stop responding again, but you can probably click the "cancel" button (to abort indexing, without quitting the indexer) before this happens. Then you should be able to access the hogs list and the blacklisted files.
From the hogs list or from the blacklist, you can double click or right click an item to reveal it in the Finder, or to open it in its application (Mail, in your case). Maybe this will reveal that some messages are malformed and contain huge amount of encoded text? If not, here is something you can do which could help to understand what is happening:
- click "update" to update your index again
- in Activity Monitor, type "crawler" in the toolbar search field to focus on FoxTrot's crawler
- when this process starts using CPU, sample it.
You can take multiple samples at different times. Examining those samples could give me some information. I forgot to ask, but are you using some Mail plugins, like MailTags, or maybe some encryption plugins?
Jérôme - CTM Engineering
------------------------------------------------------------ ---------
"With FoxTrot I find files I do not find with Spotlight.
With FoxTrot I do not get swamped with 2000 found files in one bunch,
under "Documents" and now I have to scroll through a long list - with
FoxTrot I can quickly and easy narrow my search.
With FoxTrot with one click I can see a preview with the search term
highlighted.
One of the very important reasons I like FoxTrot is that it's results
seem so much more appropriate.
And searches in FoxTrot returns are almost instantly - way way faster
than in Spotlight"
Marlyse Comte, FoxTrot Personal Search user
Download a demo version from www.foxtrot.ch
------------------------------------------------------------ ---------
Jérôme - FoxTrot Engineering
|
|
|
|
| Re: Indexing Apple Email runs very slowly [message #485 is a reply to message #477] |
Mon, 20 February 2017 22:40 |
Robert White
Messages: 6
Registered: February 2017
|
Junior Member |
|
|
I have attached about 5 samples from Crawler. I deleted the old Mail index
and set up a new one with the predefined template. The samples were done
at the start, after 10,000 files were indexed, then 20,000 and so forth
with the last one after it had been running several hours. The whole
process bogged down shortly after 20,000 files appeared to have been
indexed (but if I try to find something using just that index I don't get a
result - so it does not appear Mail is even partially indexed.
I also checked for plug-ins. I only have Mail Act-On. I did note that
there were a bunch of inactive Mail Act-On related "bundles" in the
User/Library/Mail folder. I deleted these before I recreated the index but
it doesn't appear to have had any impact.
On Thursday, February 16, 2017 at 11:16:26 AM UTC-8, Robert White wrote:
>
> I set up an index for Apple Mail using the predefined location. I have it
> set to index content. It starts indexing OK (e.g. it moves along pretty
> quickly) but then it starts to run slower and slower and eventually appears
> to freeze. I let it run for two days and it still had not completed. I am
> not sure how many email messages there are - probably close to 100,000. I
> am not sure if there is something the User\Library\Mail folder that is
> hanging it up. Any advice? Thanks
>
|
|
|
|
| Re: Indexing Apple Email runs very slowly [message #520 is a reply to message #477] |
Fri, 31 March 2017 01:08 |
Robert White
Messages: 6
Registered: February 2017
|
Junior Member |
|
|
Sorry for the long delay on getting back to the group on this issue. After
working extensively with FoxTrot (who provided great support) it appears
the problem is as follows.
Any MIME format email with no content will apparently effectively bring
FoxTrot Pro to a halt (or at least very, very, very long database updates).
Any MIME format email with content can be indexed but can slow down the
index updating as a result of a lot of long text strings in them.
I am not sure how I obtained so many MIME format emails with no content.
At some point I moved from Windows (and Outlook for Windows) to a Mac with
Apple Mail and Outlook for Mac. It was likely the result of "Moving"
emails from the Exchange Server folders under Outlook for Windows to the
Mac but it was a few years ago and I don't remember the steps I took.
Below is the email I sent to Foxtrot to tie off the issue:
I just wanted to close the loop with you on the status of this.
"I was able to find and isolate all MIME format emails with no content.
I deleted these.
I then isolated all MIMe format emails with content and moved them to a
separate folder under User\Library\Mail\V4. I then created an index for
this folder.
I updated all Indexes. They all updated - albeit the one with the MIME
format emails took awhile - I guess because it has emails with all those
very long text strings with what appear to be random characters.
So at this point all is working well. The only thing I think I should
avoid doing is updating the Index with the MIME format emails. Because
these tend to be very old emails, there doesn’t appear to be any reason
to update it. I think the issue was created when I moved from Outlook
to Apple Mail and in the process of transferring over some emails for
some reason they were converted to MIME format. These are emails that
were saved to my computer and don’t reside on a mail server anywhere.
At some point I will search for emails with the text “MIME” in all
folders other than this one I created just for MIME format emails to see
if any recent emails have that issue. If they do, then there is
something about moving emails from say the Inbox to a folder On My Mac.
Will let you know if I find anything of interest."
On Thursday, February 16, 2017 at 11:16:26 AM UTC-8, Robert White wrote:
>
> I set up an index for Apple Mail using the predefined location. I have it
> set to index content. It starts indexing OK (e.g. it moves along pretty
> quickly) but then it starts to run slower and slower and eventually appears
> to freeze. I let it run for two days and it still had not completed. I am
> not sure how many email messages there are - probably close to 100,000. I
> am not sure if there is something the User\Library\Mail folder that is
> hanging it up. Any advice? Thanks
>
--
---
|
|
|
|
| Re: Indexing Apple Email runs very slowly [message #521 is a reply to message #520] |
Tue, 11 April 2017 11:15 |
FoxTrot Engineering
Messages: 427
Registered: April 2020
|
Senior Member |
|
|
Robert White wrote:
> Any MIME format email with no content will apparently effectively bring
> FoxTrot Pro to a halt (or at least very, very, very long database updates).
To be exact, you are referring to messages with no *visible* content when displayed in Mail.app; but these are in fact messages with a corrupted MIME header. If they contain large attached files, FoxTrot handles the encoded attachment as if it was plain text, and in this situation, indexing can become abnormally slow.
Jérôme - CTM Engineering
------------------------------------------------------------ ---------
"I'd been using Entourage X ever since switching to Macs 2 years ago
and was growing frustrated with its slow searches and increasing
instability as my database grew. Decided to look for an alternative
and tried Mail, MailSmith, Thunderbird and PowerMail. Settled on
PowerMail and am extremely happy with this application."
PowerMail user comment on www.versiontracker.com
Download a demo version from www.ctmdev.com
------------------------------------------------------------ ---------
Jérôme - FoxTrot Engineering
|
|
|
|
Goto Forum:
Current Time: Sat Nov 01 00:34:42 GMT+1 2025
|
] 
FoxTrot Search Forum
Search
Help
Members
Register
Login
Home
