The user performs search and retrieval of multimedia content using as query text, images, 3D objects and their combinations. The search engine is available at the following URL:
The user can log in to the system (with a user account) or not. In this experiment, no log-in is required.
1.1. Text-based search vs. Image-based search
In order to implement this scenario, you can use the sample queries that are available at the following link: http://dl.dropbox.com/u/80197032/Sample_Queries_1.zip
Here we are searching for a specific media item (e.g. the 2D image of a flower). Here, the case is that the user takes a picture of a flower. Although he/she is not aware of its name, he/she would like to get some additional information about it (plus additional photos, 3D objects - if any, videos, etc.). The first keyword that comes to mind is “flower” (since the specific name of the flower is not known). Type the relevant keyword (“flower”) to the edit box of the GUI and press the button “Search”. The engine returns results from the database.
- Search by Image: Press the button. Drag and drop or upload the query image (one of the samples provided in “Sample_Queries_1” folder). Press the search button. The engine returns results with visual similarity to the given image.
Compare the results of the text-based query to those returned using an image query. The comparison should be done according to which result list is more satisfactory.
1.2. Multimodal search
In order to implement this scenario, please download the sample queries that are available at the following link:
In this case, more than one queries are combined.
- Search by Image: Press the button. Drag and drop or upload the query image (car_4.jpg) from the “Sample_queries_2” folder. Press the search button. The engine returns results with visual similarity to the given image.
- Search by 3D object: Press the button. Drag and drop or upload the query 3D object (car_3d_1.dae) from the “Sample_queries_2” folder. Press the search button. The engine returns results with visual similarity to the given 3D object. A screenshot of the 3D object is also available if the user has no 3D viewer.
- 3D object + image: first enter as query a 3D object. Then enter as query an image. Press the search button. The engine returns results with visual similarity to both the given 3D object and the image.
Compare the retrieved results with the multimodal queries with those returned with mono-modal queries (using one media each time).
1.3. Free search tasks
Download the “Sample_queries_3” folder from the following link:
Make similar tests with mono-modal and multimodal queries from the Sample_queries_3 folder (text, image, 3D object and their combinations). Evaluate the retrieved results (if they are relevant to the given queries).
2.1. Results Presentation
After pressing the “Search” button, a number of relevant items is retrieved. If you press the mouse (left-click) on one of the results, a pop-up box appears. It includes:
- Image of the object
- 3D representation of the object (if any)
- Sound of the object (if any)
- Video of the object (if any)
The results are shorted by relevance to the query.
3.1. Refine Search
Two options for refinement of search results are provided:
- The “Find Similar” option: The user selects one of the retrieved results and presses the link “Find Similar”. A new search is initiated using as query the selected object.
- The “Relevance Feedback” option: The user marks one or more relevant results by pressing the “star” button (bottom right corner of the object’s visualisation box). After selection of relevant results, the user presses the “Search” button, i.e. the “>” button.