Scripture Indexing tool beta available

From here you can check the latest version of the software and related apps (e.g. Importer utillity). New add-on modules will be announced here also (Bibles, commentaries, dictionaries, etc)
Pastor.Baldwin
Posts: 29
Joined: Fri Feb 19, 2010 6:25 am
Location: Idaho
Contact:

Scripture Indexing tool beta available

Post by Pastor.Baldwin »

What it is:
A tool that reads twm book files, finds every hyperlinked scripture references and builds a commentary file that will list all the books that a verse is found in. Clicking on the link in the commentary instantly takes you to the bookmark for that scripture reference. See the screen-shot below. (In the screen shot top right is the "Index" commentary created by the program and lower right is the book opened to the link that was clicked on.)
[it can optionally turn scripture references with Roman Numeral Chapters into verse hyperlinks as theWord doesn't yet do that.]

What I need:
A beta tester with a good knowledge of computers and the patience of a saint.

How to use it:
steps (quite tedious!)
  • 1) backup all your files!
    2) for each book you want included in the index you MUST prepare it as follows:
    • a) unlock the module if it is locked
      b) set the title name to something useful (I recommend "Author - Title" so that books by the same author are listed together.)
      c) detect all verses
      d) make it "uncompressed" if it is compressed
      e) convert it to rtf format
      f) if it was locked you can now re-lock it if desired
    3) copy the book files to another directory (Not a subdirectory of the official theWord directory) and put the program in that location
    4) run it and wait... (took 3 hours for 150 books, some of which were in Roman Numerals)
    5) copy the index.cmt.twm and the modified books to the book directory used by theWord on your system (NB: Make sure you have backups of the original files!)
    enjoy
What it actually does:
  • 1) it optionally hyperlinks scripture references with Roman Numeral Chapters
    2) It searches for hyperlinked verses and inserts a twm Bookmark in front of them [NB: Modifying the original book file] and writes the verse reference, book title and topic into a temporary database table. The topic is hyperlinked to the bookmark so that when you click on it you are taken to the location in the book. (verse ranges are expanded to multiple entries, chapters are not.)
    3) after all files have been modified it goes through the temporary table and builds a commentary entry for each verse found.
Let me know if you are interested in beta testing. The tool is a simple tcl/tk script I wrote for myself and not worth making more sophisticated since something like this will eventually be built into theWord but I am willing to share it in the interim.
Attachments
Screen Shot
Screen Shot
TWMindexer.gif (209.23 KiB) Viewed 16640 times
Last edited by Pastor.Baldwin on Thu Jun 03, 2010 2:39 am, edited 1 time in total.
Pastor Mark Baldwin
Missionary to Cambodia
UBC4ME
Posts: 178
Joined: Mon Nov 02, 2009 2:43 am
Location: Greenville, South Carolina

Re: Scripture Indexing tool beta available

Post by UBC4ME »

I REALLY like this idea. How many times have we all looked in the index of a book to see if the author has referenced a specific verse. How many times have we looked in the index to see if the author has referenced a specific verse only to find the author/publisher did not include a scripture reference! Nice work, if i had the needed time to give to this project I would very much like to be involved. This one will be have a great ROI. Pastor.Baldwin, thank you for this effort.
Dave
Gods gifts comes wrapped many different ways.
eyermatt5683
Posts: 30
Joined: Thu Nov 19, 2009 9:46 am

Re: Scripture Indexing tool beta available

Post by eyermatt5683 »

I would love to beta test this. Please reply.
Pastor.Baldwin
Posts: 29
Joined: Fri Feb 19, 2010 6:25 am
Location: Idaho
Contact:

Scripture Indexing tool is available

Post by Pastor.Baldwin »

Well... The code runs and does for the tester what it does for me so I'll make the link public. Enjoy. Be sure to follow the instructions in the initial post or it won't do anything for you. If there is interest and need I will flesh out those details in a users guide. (But my time is preciously limited right now.)

I'll add one more note here to reiterate and explain:

You need to prepare the files for indexing. This is annoying and tedious but necessary and the final reward is very great if you have lots of books! If you don't do this the book may be skipped or no entry's will be gleaned from it. You must check the user module box if not checked, detect all verse references since they're often not written in the file yet but "Auto detected" on display. (My tool looks for them to be detected already and saved as hyper-links in the file) then you must "convert module content to rtf" since I can't read the "native" rvf format. (those are both done from the drop down menu in the "settings/actions" tab of the module properties dialog for the module.) Otherwise nothing is detected. (Don't forget to make sure the "Module is compressed..." box is unchecked also.) NB: if you change anything, even 1 character, the section will be automatically converted to rvf format and you will need to convert it to rtf again.

You must copy these newly modified modules to the "work" directory where the tool is and you must copy them back to the books directory as a "bookmark" will have been added by my tool to each and every verse reference in the module.

God Bless and enjoy!

The tool is at:
http://www.divshare.com/download/11553955-9a4

Please post feedback and questions here.
Pastor Mark Baldwin
Missionary to Cambodia
User avatar
JG
Posts: 4599
Joined: Wed Jun 04, 2008 8:34 pm

Re: Scripture Indexing tool beta available

Post by JG »

Hi Mark, just some feedback.
I have just tried a single module from the TW site, American Tract Dictionary. The index produced does not appear to get any NT entries; but I can see the bookmarks that the indexer has inserted in the text at NT refs. Can you confirm an issue?
Jon
the
Word 6 Bible Software
OS for testing; Windows 10
Beta Download ------Beta Setup Guide------On-line Manual------Tech doc's and Utilities------Copyright Factsheet
Pastor.Baldwin
Posts: 29
Joined: Fri Feb 19, 2010 6:25 am
Location: Idaho
Contact:

Re: Scripture Indexing tool beta available

Post by Pastor.Baldwin »

JG;

I had been having a problem with images in the rtf code causing TCL (the scripting language the tool is written in) to hang. The copy of the tool you have is set to skip any entry with a picture in it. That module you reference has many images in it and comes up 2000 verses shorter if I skip sections with embedded images. I have since found a workaround for this problem and re-enabled indexing entries with images in my test code. I've experienced no hangs so I've released V1.1 to the site which will no longer ignore sections with images.

Pastor Baldwin

Download folder:
http://www.divshare.com/folder/725927-14c
Pastor Mark Baldwin
Missionary to Cambodia
User avatar
JG
Posts: 4599
Joined: Wed Jun 04, 2008 8:34 pm

Re: Scripture Indexing tool beta available

Post by JG »

Hi Mark, I am sorry to report that this is still not working properly with the American Tract Dictionary. The last entry in my index is at Psalms. It looks like there is more to the issue than you thought.
Jon
the
Word 6 Bible Software
OS for testing; Windows 10
Beta Download ------Beta Setup Guide------On-line Manual------Tech doc's and Utilities------Copyright Factsheet
Pastor.Baldwin
Posts: 29
Joined: Fri Feb 19, 2010 6:25 am
Location: Idaho
Contact:

Re: Scripture Indexing tool beta available

Post by Pastor.Baldwin »

hmmm... That's a tough one. The original (V1) code produced some NT links. the new one produces many thousands of NT links and looks to cover every book in the NT.

Matthew 1:1 has
A Dictionary of Holy Bible (AmTract)
- Book
- Genealogy
- Generation
- Joseph
- Mary
I'm at a loss for what your issue might be. could you have an odd configuration? Are you using the latest version? (some hyperlink stuff was changed recently. Does theWord give an Error when it starts?

Please check the out.txt file and see if there are any errors other than the common "verse range" errors. If you know how to open a SQLlite file (using something like http://www.sqliteexpert.com/) you could also look at the "bible_refs" table and see if there are any books ("bi") of 40 or more.

My file for that one book is 30mb but when I "Prepare module for distribution" it becomes 20mb and zip that file it's only 800K. 800k could probably make it as an email to Cambodia. I'll PM you my email if you promise not to send me anything more than about 1mb. My internet here is horrible and big emails can clog things up good. You could zip both the index.cmt.twm & out.txt file and email them and I'll look them over.

Sorry for your aggravation. I work hard to minimize that to other people. In the end this tool is worth the trouble to me, and I hope, to others.
Pastor Baldwin
Pastor Mark Baldwin
Missionary to Cambodia
User avatar
JG
Posts: 4599
Joined: Wed Jun 04, 2008 8:34 pm

Re: Scripture Indexing tool beta available

Post by JG »

Hi Mark,

I have found the conditions for me that cause indexer to fail, it is when I use Vista. I had accurate results when I used, indexer in a XP virtual machine.
Please don't waste any of your time trying to figure this on my account, I was just giving it a quick test for you. I am quite happy to wait for TW to develop in this area.
Jon
the
Word 6 Bible Software
OS for testing; Windows 10
Beta Download ------Beta Setup Guide------On-line Manual------Tech doc's and Utilities------Copyright Factsheet
Manuel
Posts: 174
Joined: Wed May 05, 2010 12:45 am
Location: Santiago, Chile

Re: Scripture Indexing tool beta available

Post by Manuel »

Dear Pastor.Baldwin:

The indexer tool works in any file twm or only in top books?
Could you put the source code in the forum? or send it to me?
Thanks in advance.

In Christ.
Manuel Dïaz
Awaiting the return of the Lord (The Glorious rapture of the Church)...

http://jesus-christ-is-coming.blogspot.com/
http://www.cristo-viene.cl
Pastor.Baldwin
Posts: 29
Joined: Fri Feb 19, 2010 6:25 am
Location: Idaho
Contact:

Re: Scripture Indexing tool beta available

Post by Pastor.Baldwin »

Manuel;

Yes, my tool should work on any .twm file. I don't understand the bible format so it won't work on those files. (Not to mention there is no need to index them as such.)

The tool is written in TCL (tool control language) which is very easy to pick up. Sadly this code was written for e-sword and grew piece by piece as I experimented and then got adapted to theWord. It is a bit of a kludge though not to bad. Still... if you're a programmer and can install tcl/tk and the sqlite plugin I could send you the code. What were you thinking of doing with it? Perhaps it's something I could do?

I may be slow to respond as I'll be traveling in Singapore and Indonesia for a few weeks and may have limited internet access.

In His Service
Mark
Pastor Mark Baldwin
Missionary to Cambodia
Manuel
Posts: 174
Joined: Wed May 05, 2010 12:45 am
Location: Santiago, Chile

Re: Scripture Indexing tool beta available

Post by Manuel »

Thanks for yor reply Pastor.Baldwin (or should I say Mark? :D )
I'm not a programmer but i'm interested in the code because i finded very usefull and i thought i could learn about this tcl/tk language. I'm very happy with the word software and your application. That's the whole issue. Also, as you probably noticed my native language is not english (sorry for bad writing or mispelling), i'm chilean so my language is spanish but in the spanish forum there's too little activity...

Since your offering maybe you can help me with this task, I have several books in english (most of the material is in that language) and there are words that i don't understand (some new, some archaic) so i have dictionaries of english (webster, english-spanish, king james, etc) do you know if there is a way that when i mark (enlight) a word this can be searched in the dict. and show results in a bubble window? (just like any bubble featured in theWord).

Maybe Costas could help us with this because i think many of us (non english speakers) would like to read some no native language books...

I hope i made my self clear enough.

Best regards.
In Christ,

Manuel.

[ I have moved the reply now to "Translate on the fly"] JG
http://forum.theword.gr/viewtopic.php?p=11817#p11817
Awaiting the return of the Lord (The Glorious rapture of the Church)...

http://jesus-christ-is-coming.blogspot.com/
http://www.cristo-viene.cl
pfpeller
Posts: 109
Joined: Sun Dec 06, 2009 7:00 pm

Re: Scripture Indexing tool beta available

Post by pfpeller »

Pastor Baldwin,

Thanks for making this tool. I have been working with it some today and it is working great. It is going to be very useful.

I was wondering if you have to redo the entire cmt file over again if you wanted to add a book to the commentary?

Blessings,
Peter
Pastor.Baldwin
Posts: 29
Joined: Fri Feb 19, 2010 6:25 am
Location: Idaho
Contact:

Re: Scripture Indexing tool beta available

Post by Pastor.Baldwin »

Hi Peter;

Thanks for the encouragement! Yes, you must rebuild from scratch every time. Because it's a 2-step tool you must do step 2 each time though I could code such that step 1 is only done for changed sections but I didn't do that. It would take a long time to write and I'm always swamped these days. I do hope to come back to the project and do a video one day and perhaps write it in c++ or something else more useful than TCL. I make great use of this in my weekly sermon & lesson prep.

Pastor Mark

(Step 1 is to find and index every scripture reference, step 2 is to build a cmt page for all the verses found for each verse in step 1.)
Pastor Mark Baldwin
Missionary to Cambodia
pfpeller
Posts: 109
Joined: Sun Dec 06, 2009 7:00 pm

Re: Scripture Indexing tool beta available

Post by pfpeller »

Hello,

Thanks for your response. I hear rumors that this functionality will be in some way available in theWord someday so what you have done is developed enough.

I was able to create thee largeindexed commentary files that are going to be very helpful to me. Thanks for your efforts.

One thing I learned is to get most of the modules I created in e sword to work correctly I had to run the "detect all verse references" and "convert to rtf" routines twice on files. I have to do this for MySword conversion anyway to get scripture links to migrate, so I had to only go back and fix some of the files.

Blessings,
Peter
Post Reply