Need a C# native client API for Sphinx Search to use in your C# or VB ASP.NET projects? So did I, so I wrote one.
Yesterday, I found Sphinx Search and decided to try implementing it in place of the (horrid) MySql Fulltext searching for my Photocore project. After downloading the binaries and indexing a couple million rows of metadata, I was amazed at how well it performs. It indexed all my data in less than a minute (compared to the 30 minutes required by MySql Fulltext) and I haven’t come up with a search that takes longer than 0.005 seconds. I was hooked immediately. So I needed a .NET API because I didn’t want to patch my database server to use the Sphinx plugin.
Source download after the jump.
Download v0.2 source | binaries
Download has no warranties, blah, blah, blah. It’s based on the Java API and is released under the GNU GPL license.
Important Note
It does not have all the features implemented. Most notably, it does not have support for Exceprts. If you modify code to make it work, send it back and I’ll post the updated version here. Or, if I get enough requests to implement it, I might. But my use of Sphinx does not include excerpts, so I didn’t bother at this point.
Other Notes
This is a pretty quick-and-dirty project, I’ll proabably clean it up over time and repost new versions, but for now make sure you test it thoroughly before putting it into production on your own sites. That’s your responsibility…
It is tested against the Sphinx searchd server version 0.9.8-dev running on both linux (ubuntu) and windows (vista) with simple queries only. I didn’t do anything fancy in my queries.
It does not work against 0.9.7 or earlier because those did not support multiple queries. It should work against future versions of Sphinx server. No promises though.
Usage
Super-simple to use after you have a searchd server running with an index. Three lines will do a search for “rose bowl gooley” matching any index (“*”) and will get the second page of results (items 41-80) with a max result count of 10,000 documents.
// set hostname and port defaults
SphinxClient cli = new SphinxClient(hostname, 3312);
// set the pagination params
// (this is page 2, 40 per page, 10000 max)
cli.SetLimits(40, 40, 10000);
// define the query and index
// ("*" means any index)
cli.AddQuery("rose bowl gooley", "*");
// run the query and get the results
SphinxResult[] results = cli.RunQueries();
The SphinxResult object contains an array of SphinxMatches which will provide you with the DocumentId numbers for all matches. Simple as that.
Happy searching!
- BROWSE / IN TIMELINE
- « Build your own modded System.Web.Extensions.dll
- » Facebook App – Game – ChopShop Racing
- BROWSE / IN dot.net software
- « Build your own modded System.Web.Extensions.dll
- » Facebook App – Game – ChopShop Racing
COMMENTS / 27 COMMENTS
Charli said on Jan 16 08 at 12:30 pmI just stumbled into your website while checking out sphinx on Google, thought someone might have implemented sphinx using a .NET language, and here it is. This is really good news, I haven’t tried your code yet but by the quality of it I’m guessing it’s working great.
We haven’t (until now) been able to implement full text search on our forum since mysql fulltext search is way to slow, and I haven’t been a fan of patching mysql with sphinx since software I patch usually tend to work real bad. I’m gonna play around with your code together with sphinx to see how it works :).
theGooley said on Jan 17 08 at 12:21 pmhey Charli -
Glad to hear that it might be useful to you. I’m going production with it this weekend and it has done well in all my tests thus far.
Let me know how it works for you (and where you’re using it, since I’m curious).
cheers!
cod_ferrow said on Mar 25 08 at 5:09 amHello sir,
there is a small problem with the attributes reading – the numbers are not real. I’ve fixed it. If anyone need these sources, I’m here.
bob yakovic said on Aug 27 08 at 1:31 pmis SphinxClient thread safe? Can I make a static SphinxClient and use it across threads?
ankesh said on Sep 11 08 at 1:15 amHi
I am trying your code but not able to connect through TCP .The message are“No connection could be made because the target machine actively refused it”
Please help me ASAP.
theGooley said on Sep 11 08 at 4:24 amhi ankesh -
that sounds like a firewall blocking your traffic or the wrong port you’re trying to use. make sure your sphinx server is at the hostname you pass to the SphinxClient constructor, and make sure the server is using the default port 3312 or change the port to match your server.
that TCP error probably has nothing to do with the client API code.
hope that helps.
miha said on Oct 04 08 at 12:05 pm[quote comment="3017"]Hello sir,
there is a small problem with the attributes reading – the numbers are not real. I’ve fixed it. If anyone need these sources, I’m here.[/quote]Can you send it to miha at studiopesec dot com.
Tnx in advance
Miha
Peter said on Dec 04 08 at 12:23 pm[quote comment="3017"]Hello sir,
there is a small problem with the attributes reading – the numbers are not real. I’ve fixed it. If anyone need these sources, I’m here.[/quote]I am also interested in this fix: peter at emblemsoftware dot com
Thanks.
Itye said on Dec 08 08 at 4:18 pmHi,
I noticed that every time I run a query, a new Tcp connection is established. Opening a new connection lasts 1000~ milliseconds.
It makes 1 second delay for querying results, and there I lost the super speed of Sphinx.
What do you recommend in order to improve performance ?Thanks,
Itye.
David said on May 12 09 at 7:44 amFor unsigned integer attributes to appear correctly you need to change:
/* handle everything else as unsigned ints */
long val = bw.ReadUInt32();
to
/* handle everything else as unsigned ints */
uint val = ReadUInt32(bw);to make it go through byte reversal. (no need to ask me for the source!). Also easy to add the latest enumerations (and change the validation code) to use options in 0.9.9.
Ben said on Jun 05 09 at 1:16 amGreat to see this project. Even better would be a GitHub project and better contributions. “I fixed some things” doesn’t help other users, unfortunately.
Mohit said on Jun 24 09 at 9:40 amHi,
I had change the port according to the server but getting error
“Connect: Read from socket failed: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.”Plz help…
and also can you please let me know how we can assure that whichever port i m using is the correct one?.Thanks
Mohit
Charlie said on Jun 30 09 at 9:38 pmHello,sir
There is a problem with getting a match’s attrValues.its always be nothing.
How can I init a match’s attrValues array.
Charlie said on Jun 30 09 at 9:42 pmMaybe my problem is the same to you.
When can the version be published?
Zeke said on Jul 08 09 at 10:23 pmWe are looking for a solution for Sphinx to run on 64bits Windows Server. Any clues? Don’t even mind a fee for a compilation that will work on 64bits windows.
Vincent Theeten said on Jul 24 09 at 3:02 amGreat initiative Christoper. Any chance you’ll be putting up a shared project somewhere? Google code? Github? I’m sure others are willing to contribute as well.
Bill said on Aug 28 09 at 1:53 pmAdded support for floats. Here’s some code:
private void WriteToStream(BinaryWriter sw, float data)
{
int intBits = FloatToInt32Bits(data);
byte[] d = BitConverter.GetBytes(intBits);
sw.Write(_Reverse(d));
}public static unsafe int FloatToInt32Bits(float value)
{
return *(((int*)&value));
}public static unsafe float Int32FloatToBits(int value)
{
return *(((float*)&value));
}
Bill said on Aug 28 09 at 2:01 pmAlso added UpdateAttributes API Call:
/**
* Connect to searchd server and update given attributes on given documents in given indexes.
* Sample code that will set group_id=123 where id=1 and group_id=456 where id=3:
*
*
* String[] attrs = new String[1];
*
* attrs[0] = “group_id”;
* long[][] values = new long[2][2];
*
* values[0] = new long[2]; values[0][0] = 1; values[0][1] = 123;
* values[1] = new long[2]; values[1][0] = 3; values[1][1] = 456;
*
* int res = cl.UpdateAttributes ( “test1″, attrs, values );
*
*
* @param index index name(s) to update; might be distributed
* @param attrs array with the names of the attributes to update
* @param values array of updates; each long[] entry must contains document ID
* in the first element, and all new attribute values in the following ones
* @return -1 on failure, amount of actually found and updated documents (might be 0) on success
*
* @throws SphinxException on invalid parameters
*/
public int UpdateAttributes ( String index, String[] attrs, long[][] values )
{
/* check args */
Debug.Assert ( !string.IsNullOrEmpty(index), “no index name provided” );
Debug.Assert ( attrs!=null && attrs.Length>0, “no attribute names provided” );
Debug.Assert ( values!=null && values.Length>0, “no update entries provided” );
for ( int i=0; i<values.Length; i++ )
{
Debug.Assert ( values[i]!=null, “update entry #” + i + ” is null” );
Debug.Assert ( values[i].Length == 1 + attrs.Length, “update entry #” + i + ” has wrong length” );
}/* build and send request */
if (_client == null || !_client.Connected)
{
this.Connect();
}
if (_client == null) return -1;
StreamReader sr = new StreamReader(_client.GetStream());
BinaryWriter sw = new BinaryWriter(_client.GetStream());
BinaryWriter tsw = new BinaryWriter(new MemoryStream()); /* temp command buffer */
try
{
WriteToStream(sw, VER_MAJOR_PROTO);
WriteToStream(sw, (short)SEARCHD_COMMAND_UPDATE);
WriteToStream(sw, (short)VER_COMMAND_UPDATE);
this.WriteToStream(tsw, index);
this.WriteToStream(tsw, attrs.Length);
for (int i = 0; i< attrs.Length; i++)
{
this.WriteToStream(tsw, attrs[i]);
}this.WriteToStream(tsw, values.Length);
for (int i = 0; i < values.Length; i++)
{
this.WriteToStream(tsw, values[i][0]); /* send docid as 64bit value */
for (int j = 1; j < values[i].Length; j++)
{
this.WriteToStream(tsw, (int)values[i][j]); /* send values as 32bit values; FIXME! what happens when they are over 2^31? */
}
}this.WriteToStream(sw, (int)tsw.BaseStream.Length); // send the request size
sw.Write(((MemoryStream)tsw.BaseStream).ToArray());
sw.Flush();} catch ( Exception e )
{
_error = “internal error: failed to build request: ” + e;
return -1;
}/* get and parse response */
byte[] response = GetResponse(_client, VER_COMMAND_UPDATE);
if ( response == null )
return -1;try
{
BinaryReader br = new BinaryReader(new MemoryStream(response));
return this.ReadInt32(br);
} catch ( Exception )
{
_error = “incomplete reply”;
return -1;
}
}
Bill said on Aug 28 09 at 2:27 pmOne more piece:
public void SetFilterFloatRange(string attribute, float min, float max, bool exclude)
{
Debug.Assert ( min<=max, “min must be less or equal to max” );
try
{
BinaryWriter bw = new BinaryWriter(_filterStreamData);
WriteToStream(bw, attribute);
WriteToStream(bw, SPH_FILTER_FLOATRANGE);
WriteToStream(bw, min);
WriteToStream(bw, max);
WriteToStream(bw, exclude ? 1 : 0);
}
catch ( Exception e )
{
Debug.Assert ( false, “IOException: ” + e.ToString() );
}
_filterCount++;
}
Shaun said on Oct 05 09 at 11:07 amHey everyone, I re-wrote a .NET client for Sphinx from scratch this past weekend and have posted it up on Google Code (http://code.google.com/p/sphinxdotnet) Feel free to join the project and help out if you want. I think it is feature complete for the sphinxsearch trunk (release 2011)
Rusted said on Oct 13 09 at 1:55 amCheck this client too http://code.google.com/p/sphinx-dotnet-client/
It’s open source and supports all latest features and Sphinx server protocols,
ADO.NET like API. Provided with samples and Sphinx GUI index test tool.
Matt said on Nov 04 10 at 2:38 pmHi there – have you published any updates to your code since this post? I’m wondering how stable you found this release. I’m going to check it out.
Question – does your API include methods to add text to the index or do I do that separately?
Thanks…
kendall said on Nov 14 10 at 10:37 pmI am attempting to use you api with VB.net however I am having just a bit of trouble. In your example I do not see where you import the
SphinxSearchApi.dll. Could you please shed a little light on this for me.
Tralamazza said on Nov 16 10 at 1:47 pmHi and thanks for this project, I wrote a simple sphinx client myself a while ago. I made it available here: dotsphinxclient.codeplex.com
armando said on Apr 28 11 at 3:34 pmHi,
Thanks for this library, I wonder if it allows real time index, I want to use that new feature, available since sphinx 1.10.
Simon said on Jan 05 12 at 1:40 pmThanks for this project – I can’t seem to get it to work though. I’m using vb.net and have translated your sample code from this page into vb.net. And get a “Object reference not set to an instance of an object” error when trying to do anything with the results array.
Maybe I’m not connecting to the host correctly (I’m using the name my PC and port 3312 with searchd listening on that same port). I’ve also tried with my local IP address. I would have thought if my connection was wrong it’d error earlier on in the code but it doesn’t even if I deliberately but an incorrect host and port.
Any pointers would be great – thanks.
SPEAK / ADD YOUR COMMENT
Comments are moderated.
