Help - Search - Members - Calendar
Full Version: New search engine?
Great War Forum > Miscellaneous > Classic Threads
Pages: 1, 2, 3, 4, 5
geoff501
A new search engine:

http://www.hut-six.co.uk/cgi-bin/search.html

Its a little limited at present:

Input text is case insensitive.
If you search for ALDER you will get ALDER, ALDERTON, CALDER etc.
Same for Regiment Number.
The indexing is limited to United Kingdom units at the moment.
The index coverage is not 100% - but is around half a million.
It is also limited to records that have a Additional Info field.
Keywords will be searched anywhere; name, regiment number, info field.
No other fields are searched at present.
Limited to 10 results!

I tried to think of a snappy name for this, but all the Google type words are already taken. I did think of calling it Monty, since it was originally written in the Python computer language (geddit?). Also it might be inspired by the photo of General M. looking at one of his Regiment's unknown soldier's burials near Vimy.
So Geoff's Search Engine it is.
Depending on the response (pat on back/F.P. #1/warm half of shandy from the tite N&D gits/moderated to oblivion), I may add some extra functionallity. I'm off down the pub.

Geoff

ps s'pose I could call it geddit
museumtom
Its very interesting and seems a great tool as you can search by number and next of kin. The only thing I would fault is that it is not yet complete. When it is I will be using it on a regular basis and it will be a goldmine for researchers. Well done whoever started it.
Tom
Paul Reed
Interesting Geoff - I see it only displays some of the overall results (eg it would only display 10 out of 112 results for a test search I did). Is there anyway to get it to show everything?
Chris Noble
Good stuff Geoff, very interesting.
Regards,
Chris.
Chris_Baker
It's great, Geoff. Now do tell: how is it working? Have you got some kind of live link into CWGC database?
Garron
Thats fantastic, I found both soliders i was looking for and all I had was an address and no name, typed in the location, (Ferndale, North,); bang both there in the 10 results.

Thats gonna be a gold mine onces its done fully, thanks

Gaz
Willywombat
I'm well impressed!

Please, please do increase the functionality as and when you are able. This is just what everyone's been waiting for and something the CWGC have been promising but not delivering for a long time.

If you can do it, why can't they?

Bob.
joseph
Well done Geoff its impressive, if you increase the functionality it will be a first class research tool.

Regards Charles
geoff501
QUOTE (Paul Reed @ Sep 29 2007, 09:11 PM) *
Interesting Geoff - I see it only displays some of the overall results (eg it would only display 10 out of 112 results for a test search I did). Is there anyway to get it to show everything?

I have increased the list size, as a temporary measure. Might need more still. Ideal soultion is to add <previous> and <next> buttons, which will need a bit more coding. I'll add it to the wish list.


QUOTE (Chris_Baker @ Sep 29 2007, 09:35 PM) *
It's great, Geoff. Now do tell: how is it working? Have you got some kind of live link into CWGC database?

There are no live demands on the CWGC website, its all done by indexing.
geoff501
QUOTE (Willywombat @ Sep 30 2007, 08:14 AM) *
I'm well impressed!

Please, please do increase the functionality as and when you are able. This is just what everyone's been waiting for and something the CWGC have been promising but not delivering for a long time.

If you can do it, why can't they?

Bob.


I think its a question of funding for them. - it has low priority. Their current design meets their requirements very well. It is not intended to be a research tool but a finding aid for relatives.
geoff501
QUOTE (joseph @ Sep 30 2007, 09:10 AM) *
Well done Geoff its impressive, if you increase the functionality it will be a first class research tool.

Regards Charles


Quite a few entries in the log file show searches for regiments. This is not operating yet - need more work on optimizing the index, but it could be added to increase functionality.
geoff501
QUOTE (geoff501 @ Sep 30 2007, 10:58 AM) *
There are no live demands on the CWGC website, its all done by indexing.


Which means the index may not be complete or 100% correct on what is in it.
MagicRat
As other Pals have pointed out, it's a fantastic research tool and I'm sure will only get better with time. Just realised that you don't even need to enter a surname to do a search - Just entered "PATMOS", the name of the road in Camberwell where my wife's great uncle lived, and back has come 10 results, most of which would have been his neighbours.

Have also searched for people living in our road in Wells, and found Corporal Herbert Allen, who died on 30th October 1917, very near Varlet Farm, where we'll be at the end of October!

So many thanks!
Paul Reed
Thanks for answering my query Geoff - it is certainly a nice piece of work.
keithfazzani
Fantastic stuff - just put my village and county in and it came up with a list of names associated with the village - not all on the memorial - some detective work now needed! Thanks for this.
Ian Riley
Excellent. Thank you. As a minor spin-off, it shows up very quickly the number of spelling errors (for whatever reason) in well-known local street names. In my village, Twiss Green has become Swiss Green and Twin Green.

Ian
MelPack
Geoff

You are a norty genius! tongue.gif

I have found links between fathers and sons killed that I never knew existed and searches of towns and villages have thrown up connections well beyond the numbers on memorials.

I would suggest that you keep this data base separate from any further forays that you make into the CWGC as a whole.

With it being confined to the casualties with additional information means that key information is not being lost in broader searches. For example, I used the keywords Royal Berkshire which produced a small number of hits that established a father and son relationship. That information would be very hard to detect if, for example, you widened the parameters so that a search would throw up all Royal Berks casualties for the CWGC as a whole.

OK, I am being greedy here cool.gif

Keep this data base with the tweaks suggested and have a separate one for the CWGC as a whole where you can search by regiment, date of death and regimental number with a wildcard eg 18207 searchable by 182**.

Geoff you're a bloody marvellous! biggrin.gif

Regards

Mel
Pighills
Just had a quick go on it and it's great - many thanks!!!!!!
shaymen
Geoff
Brilliant mate
Search by keyword - marvellous
Glyn
MartH
It is rather fine.

Mart
John Duncan
I had a bash and it threw up another couple of men with local connections that I knew nothing about. A very useful tool for research it would appear.

smile.gif John
museumtom
I had another go at it and its fantastic!!! Well done. Lots of new information in it and faster than the CWGC.
Give yourself a great big pat on the back, you deserve it.
Regards.
Tom.
geoff501
QUOTE (museumtom @ Sep 30 2007, 04:42 PM) *
Give yourself a great big pat on the back, you deserve it.


Better than a half of warm shandy - thanks.

Over 700 searches carried out in the last 24 hours!

Geoff
Phillip Roder
Great Job, discovered dead relatives I never knew I had.

More work on the Genealogy front needed by self (just when I thought my familytree was almost finished!)

Frustrating that the only relative, KIA, date, service number, memorial etc that I knew was not there, however you did say it was not complete.

Does that mean the body was never found?

Will use it to check my familytree now as I stopped most of my grand uncles research at the 1901 census...that will explain why there are people that lived in the house not on my data base.

Thanks for the help.

Phil
Greyhound
Thanks Geoff! Had a quick look and found a couple more entries of interest.

This could prove very useful indeed. Brilliant stuff.
charlielavin
Bloomin' wonderful!

Congrats
MartH
So clever its the sort of site, GW ones would love to link to, has anyone asked you yet?

Regards

Mart
geoff501
QUOTE (MartH @ Oct 1 2007, 05:00 PM) *
So clever its the sort of site, GW ones would love to link to, has anyone asked you yet?


Its a bit embryonic at the moment, I've not sought any linking. I have an update that may go up later this evening.
Also still need to work on regiment indexing.

geoff
MartH
Its a quality search tool, and would be so useful for many GW sites researching people, I'm surprised ancestry.com have not contacted you.

Best regards

Mart

PS did you come to the database conference in March?
geoff501
QUOTE (MartH @ Oct 1 2007, 06:15 PM) *
I'm surprised ancestry.com have not contacted you.


Perhaps you think they should hire me to sort out the pension mess smile.gif

QUOTE
PS did you come to the database conference in March?


What am I, some sort of anorak! wacko.gif - I was probably there in spirit if not body.
MartH
Well you should have been there at the Database conference, you input would have been valuable and it was very interesting re data sources

Joking apart, this sort of search engine for www.ancestry.com would be commercially interesting.

It is the obvious way to search CWGC, I'm just scared you server will go under, check how your billed please.


Regards

Mart





PS you you sort out the German equivalent too?
museumtom
Will you be able to cover ww2 also?
regards.
Tom.
geoff501
QUOTE (Paul Reed @ Sep 29 2007, 09:11 PM) *
Interesting Geoff - I see it only displays some of the overall results (eg it would only display 10 out of 112 results for a test search I did). Is there anyway to get it to show everything?


I've added some paging to the output. It's not ideal in that it does not allow jump to numbered pages and you don't know how many until you reach the last page. Also not sure what happens if there are a multiple of 20 records found! However I should tidy this up later, for now I want to look at the regiment index. I have also added an option to search on name beginning with or included (it defaults to include if not selected, I think - I forgot to pre-select the button).

Anyone who's previously used this may have to hit the refresh button on their browser to get the new version.

It's running slow now, I suspect the server is busy, but there's probably scope for speeding it up a little when the proper paging is in place.

Here's a reminder of the URL:

http://www.hut-six.co.uk/cgi-bin/search.html


Geoff
geoff501
QUOTE (MartH @ Oct 1 2007, 07:32 PM) *
Well you should have been there at the Database conference, you input would have been valuable and it was very interesting re data sources


To be honest, I'm not a database expert and would struggle to design one. I've seen some of their work here and admire their committment to these large projects (well done N&D ers!).

QUOTE
I'm just scared you server will go under, check how your billed please.


Paid in advance, 99.9% uptime, backed up daily - sleep well.
If the server gets overloaded, I might get problems!



QUOTE (museumtom @ Oct 1 2007, 09:04 PM) *
Will you be able to cover ww2 also?
regards.
Tom.


Tom,

Possibly, but main priority is WW1.
geoff501
Ooooopps.... There may be a bug in the update version; the fields below name are not carried forward when next page is selected, so it searches on empty fields.

geoff

edit: If a field is blank, then any fields below are ignored on page 2 onwards. Page 1 is OK. - I'll investigate!
museumtom
Geoff this is really something but could you possibly give us (doddery ould codgers) some hints to use it better please? i.e give us some examples of different types of searches. How to use the keywords 1 and 2.
Regards.
Tom.
geoff501
QUOTE (geoff501 @ Oct 1 2007, 09:30 PM) *
Ooooopps.... There may be a bug in the update version; the fields below name are not carried forward when next page is selected, so it searches on empty fields.

geoff

edit: If a field is blank, then any fields below are ignored on page 2 onwards. Page 1 is OK. - I'll investigate!


Sorry for the glitch - seems to all be in order now. (yawn)

geoff
geoff501
QUOTE (museumtom @ Oct 1 2007, 09:46 PM) *
Geoff this is really something but could you possibly give us (doddery ould codgers) some hints to use it better please? i.e give us some examples of different types of searches. How to use the keywords 1 and 2.
Regards.
Tom.


Any field is optional. If left blank it will let everything in that field through.

Enter lastname name. Select 'name begins with' or 'name includes'. as required. Will find any match on the entered name.
for example:
name begins + SMITH will find SMITH, SMITHIES, etc.
name includes + ALDER will find ALDER, ALDERTON, CALDER, etc.

Enter date (can be dd/mm/yyyy, mm/yyyy, or yyyy) First example matches date exactly, the second matches any day and the third matches any day or month, for the given year.

Enter reg number. This will find any match ie entering 123 will find 1234, 71234, G/123 etc.

The keyword operates anywhere on the name, reg number and info field - really it should be restricted to the info field and I'll fix this later. Only records that match the keyword(s) (anywhere) will be found.

Another example to try: leave all blank except keyword1 = FIVE and keyword2 = BROTHERS.

Just experiment!

Cheers,

(doddery old softweare engineer, way past his bedtime - time for cocoa)
djcrtoye
Hi tried your search enigne. Put in my home town and while scrolling through it , I found a gg uncle who I hadn't found any information on. His name was Pte Daniel Hendry Army Service Corps S/21124 28th (Labour) Coy Labour Corps, transfered to 711th Coy 349429 date of death 21-10-1917 buried in Salonika(Lembert Road) Military Cemetery. R.I.P. Thanks for letting us use it.
MartH
Hi Geoff

Knowing how people search on regiments, any chance of a separate field of that?

Regards

Mart
hywyn
Geoff
May I add my thanks and congratulations re an excellent tool. I have been searching many placenames this afternoon and like others have had a number of hits that i would not have had.
Thanks

Hywyn
Siege Gunner
Geoff, I think we should institute a 'GWF Gold Medal' for outstanding contributions by a Member to the advancement of popular research into the Great War — and you should get one. I've already discovered a number of things I wouldn't have found otherwise, and I haven't even begun to explore the full potential of your search engine yet. Many thanks, congratulations, and please continue expanding and refining it.

Mick
westkent78
Superb work Geoff.

Look forward to seeing the next version with regimental search capability. I've been missing that capability from the CWGC ever since they pulled it.

Does your gui work with any wildcards?

I second the motion to award a gold medal.
Best regards,
Matthew
geoff501
I've just posted a minor update:

Added the option to search on exact name.
Removed the keyword search on name - only applied to info now.
Added small help file.

Please remember to hit refresh, if you've used it before. You should see version 1.0.5
No wildcards yet.
No regiment yet (soon). A small number of users are still attempting to search on regiments.
Also no ANZAC or Canadian etc. I kept the system smaller to start with. They can be added.

SG, what a nice thought about the medal, and a nice idea anyway. However there are many more deserving cases before me - I'm still crawling towards the wire.

geoff
geoff501
Regiment search now added!
The index is still limited to those United Kingdom units with additional info data - about 500,000 records.
Don't forget to hit refresh on your browser.

geoff
MartH
What a star

Mart
stiletto_33853
Geoff,
This is good stuff, well done.

Andy
geoff501
QUOTE (MartH @ Oct 2 2007, 05:25 PM) *
Knowing how people search on regiments, any chance of a separate field of that?


I can see why CWGC removed the regiment search feature (although I never saw it). It can be a minefield, and probably caused more problems than it solved.
Entering a unique phrase to identify a unit is not always easy, unless you are looking for something like Buffs. I think what is needed is a keyword search on regiment in addition to the entered regiment name. There are over 400 UK 'regiments' and over 1000 in total, so a drop down selector would not be an easy option to use.
Alan_J
I'll just add my voice to those expressing similar sentiments - an extremely useful tool indeed!

Alan
ypres1418
What a boon,
have been on the site now since 25 past ten!

found lots of lads from Warrington and Seaford!

more work to do now! Thanks,

Mandy
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.
Invision Power Board © 2001-2009 Invision Power Services, Inc.