1. Abstract *

2. Introduction *

3. Background *

4. Searching Issues Today *

4.1 Text Searching *

4.2 Video Searching *

4.3 Audio Searching *

4.4 Search Engines Today *

5. The Future of Searching *

5.1 Text Searching *

5.2 Video Searching *

5.3 Audio Searching *

5.4 Geographical Searching *

6.Conclusion *

7.References *

In this paper, I plan to explore the topic of searching on the Internet, specifically with multimedia. After a brief introduction and some background information on the various indexing options for searching, this paper will explore some of the current issues regarding text, video, image, and audio searching on the Internet. Some of the issues discussed will be regarding how to approach video and audio searching by content and not based on file extensions as well as some of the better-known issues regarding text searching due to the ambiguity of the English language. The final section of this paper will present some actual case studies of Universities and private companies that are doing work on developing methodologies to search video and audio on the Web by content. Specifically, the methodologies discussed will include searching video using an animated sketch, searching video sequences using a storyboard, and searching sound based on its content and color-coded sound wave files.

The need to search through large amounts of information is something that has been around all through out history, even before the Internet. For years searching was done manually through large numbers of books and documents with the only tool being one’s eyes. Later on, the card catalog was invented and for years people could go to any library to look for a book based on their title, author, and/or subject without having to go through each shelf in the library and look at each book one by one. The invention of the card catalog provided a quick way to scan through large amounts of information to quickly find the relevant information.

With the advent of the World Wide Web, the amount of information available has significantly grown and is growing every day. The Internet has given people all over the world a way of access information from anywhere regarding any subject matter under the sun. Since there is such a vast amount of information available, there is a great need for users to be able to go through large amount of information in an automated fashion to find what they are looking for. The Internet comes equipped with a vast amount of searching tools (e.g., Yahoo, AltaVista, Excite) that allow the user to search through web pages based on specific criteria. These pages are documents from all over the world that the authors have put on the World Wide Web for all to benefit from their work.

Typically, all searching tools (e.g., library card catalog) provide users with the ability to search through text documents to find what they were looking for. Much of the search engines available today on the Internet are based on the same principle. The user enters a text word or phrase and the engine searches the registered web pages for as close a match as possible to the users criteria. However, since the Internet is an electronic computerized medium, the possibility for using the world wide web for searching multimedia is there, but is currently still in development.

As Shneiderman points out, searching through multimedia videos and audio clips is now a real challenge for the search engine developers. Given the size of the Internet, all search engines must be smart enough to evaluate a user’s criteria and bring back only the relevant video and/or audio information the user is requesting. This is quite a difficult task given the overall size of the Internet and all the information available in it. This is an area that is still in development and that one-day be available to all of us via the World Wide Web.

In order to do an electronic search of any kind, the body of information that is going to be searched has to be classified in some fashion. This classification scheme is typically referred to as an index. An index is defined as "set of methods whereby we classify information in some structured, abstract and compact form in order to find the information at some later period of time." (Turoff, Draft Book, Ch 5). Once all the information is classified into an index scheme, then the search tools will be able to look through large quantities of information in a systematic manner. There are several types of standard indexing schemes that are used by the most popular search engines on the web today.

The first scheme is the hierarchical indexing schema, which is a tree-structured system to classify objects. In this system, each object can only be in one place on the tree. An object cannot be classified in multiple spots on the tree. An application that uses the hierarchical classification system is Yahoo. Yahoo categorizes all of it web sites into headings such as Entertainment, Business, Recreations, and so on. These overall headings are then further sub-classified into sub-headings. This hierarchical classification is suited for Yahoo because it allows the user to narrow down their search into only the specific area(s) they are interested in. The only problem with this system is that as more and more site(s) become a part of Yahoo, it will be more difficult to consistently classify them into Yahoo’s already established hierarchy.

The next scheme is the subject heading indexing scheme where a set of subject headings, not related to each other, are established. All objects are then classified under these headings. This scheme allows for objects to be placed under multiple subject headings for easier searching. However, in order to properly classify object under these often-broad headings, the headings must be clearly defined.

In the key words scheme, any combination of words or phrases can be used to identify and categorize a particular object. Each object ideally should have its own set of keys, so that they can all be uniquely identifiable. The key words or phrases assigned to each object can be either free form where the user chooses the appropriate key words, or from an already defined list of options. The NJIT EIES conferencing system is an example of an application that uses the free form version of key word indexing. Each comment posted in a conference can have a set of key words to identify it. The only issue with this scheme on is that the user may not always be required to put in key words to identify the comment. If the user fails to enter key words, the object goes unidentified, and irretrievable via searching.

Facet indexing is when objects are characterized according to their attributes or a set of dimensions. For example, metals can be characterized by their color, texture, and resistance to heat. An example of this classification is the Communications of the ACM as it characterizes all its papers and articles based on their attributes. These include author, date written, publisher, volume, and number of pages.

Natural language is the idea of classifying all objects by a textual description, or abstract. All objects would simply be classified by a short description summarizing the object written by the object creator. These abstracts would not appear in any pre-defined structure, but rather free form containing whatever was written about the object.

Each of the standard indexing approaches already described can be measured in terms of performance. Often, this is used to assist the developer of a search tool in choosing a searching index scheme. There are several characteristics to consider when measuring performance:

The characteristics described above are simply one manner in which index performance can be measured. It can be used to assist the developer of a search tool to choose a particular indexing scheme. Another methodology for index performance is based on the concepts of precision, recall, specificity, and search efficiency.

Often when one is searching, the results that the searching tools give back match the criteria specified, but much additional information that is not relevant appears in the search result. Also, missing information that may also be relevant to the criteria specified may not be included in the search results. It is these relationships that precision, recall, specificity, and search efficiency measure. Precision is the measure of how much of the search results were relevant to the criteria that were specified. Recall is the measure of how much of the relevant information based on the search criteria was included in the results. In other words, recall answers the question of "do the search results include all the relevant information for that criteria available in the database." Specificity involves how much non-relevant information exists in relation to the total size of the database. Search efficiency is the overall measurement of recall multiplied by specificity.

These criteria are used to measure searching tools and the various indexing schemes involved in them. This is another approach to measure indexing and searching performance when choosing an indexing scheme as well as when developing and testing the chosen scheme in the user’s environment.

When developing Internet search engines or other searching tools, the indexing scheme that is chosen to classify information is very important. The developers should be aware of their end users and how they currently classify information. "In any application today, the process of searching and finding information is a standard, common, and growing problem. It behooves the designer to understand how humans go about classifying information into some sort of indexing schema. In many cases, this is very different from the ways that are used internally in the computer to index information." (Turoff, Draft Book, Ch 5) Regardless of the approach taken to choose an indexing scheme and measuring its performance, the overall goal of indexing is to provide a structure to the body of information that is going to be searched frequently. Essentially, the indexing scheme chosen should be one that can be familiar to the user and easy for them to use. It should fit their current environment, even if the method chosen is not one that the developer normally uses.

Given the increasingly large amounts of information available on the World Wide Web, there are many issues that come up regarding searching and the current search tools that are available on the World Wide Web. Every search engine available today provides a way to look at large amounts of data on the web, and some search engines function better than others, depending on personal experiences using them. All the search engines available, however, have the same basic process for retrieving data.

Every search engine on the Internet has the three components described above and the same basic process for indexing data to be searched. The issues regarding searching, however, also arise in the retrieval process.

Text searching is the most common form of searching today. All of the search engines available on the web have been primarily designed for searching text and retrieval is based on when the text found matches or closely matches the user’s query. The most common user criticism of text searching is getting the "extra links" that are totally irrelevant to the desired information. "Users of the Web mostly employ search engines to find the information they are looking for, but usually far too many Web addresses are returned in response to their queries. Even worse, many of the pages corresponding to these addresses are totally irrelevant to them. Technically speaking, the precision (in this context the proportion of returned addresses that are relevant) is usually very low." (Kaindl, 217) One of the reasons why the additional links appear is that most search engines today are not sophisticated enough to deal with the ambiguity of the English language. The search engines are simply trying to match the "symbols" entered as part of a user’s query to the "symbols" on the web pages. This is why when the user tries to search for information on the Magic Johnson the basketball player; they may also get links for information on Johnson & Johnson the corporation. Since the search engine is matching symbols, the links for Johnson & Johnson are a match based on the user’s criteria.

Another big issues or problem regarding searching is that more often than not, the average user does not really have a specific concrete idea of what they are looking for.

The usual experience regarding searching on the Web is that the user typically enters the search process only with an idea of the desired information. Often, the user is simply looking for information on a general topic such as "cars" or "homes." As the user searches the Web, the criteria gets more specific and refined until it becomes phrases such as "Cars built in the USA between 1951 – 1955" or "Homes up for sale in Rahway, New Jersey in 1999." Today, finding desired information on the Web is not a one-search process. It involves multiple searches with various criteria options in order to find the desired information. Since there is just so much information available on the Internet, it would be virtually impossible to find all relevant information based on the first set of criteria. Efficient searching on the web involves having to refine one’s criteria and search multiple times using the same search engines or multiple search engines. Just as searching for information in a library is multiple search process, so is searching on the Web. With so much information available, it is simply not possible to find everything with only one search.

Searching through video and/or still images presents an interesting challenge to search engine developers. The way most search engines operate today is by appending text descriptions to the video clips and/or images, so that the searches are based on the text. This enables one search on the text description of "Mona Lisa" and get back the picture, but does not enable one to search on all videos dealing with soldier marching techniques. Video and photo searching is something that is still being developed and explored. For example, being able to search for an image of the Statue of Liberty would be very challenging for developers, as this would require a query by image content. Basically, this means that the search engine or tool would have to be sophisticated enough to recognize an image of the Lady Liberty and differentiate it from all other possible images. Instead, one of the approaches for searching images is to search for distinctive features of an image. For example, the search tool could look for images with Lady Liberty’s features such as a torch or the seven spikes in the crown. Another approach would be to search for distinctive colors of the known image. In this case, the search engine could look for the distinctive fading green color of the statue.

In either case, these are merely theoretical approaches that would have to be developed. For example, searching for a distinctive feature of a torch could return hundreds of images of the use of torches in ancient times. Searching for a feature of a crown could bring back images of kings and queens of the past. In addition, in order to enter criteria for this type of search tool, the user would have to draw the desired object or feature. This would present an interesting challenge for users and developers to establish a good interface for a searching tool of this type.

Efficiently searching video is even more complex than still images because now that search engines or tools have to be sophisticated enough to handle movement, lighting, and different camera angles. The searching of a video or film would have to be more sophisticated than to simply search a video frame by frame for the desired result. Users may also want to search for specific scenes in video or for zooming in and out.

In addition, there would have to be an interface developed to facilitate this type of searching. Somehow, an interface would have to be set up to allow the user to specify specific scenes in a video, specific lighting, the zooming in and out, and specific movement. All these present interesting challenges for the developers to how to provide an easy interface for users to use when searching video.

Audio searching today is also based on the idea of appended text. Typically audio clips or audio files are appended with a text description, so that the user can query by the appended description. This enables one to search on the text description of Mozart’s Symphonies and get back the music file, but does not able one to search on all music dealing primarily pianos and/or violins.

The searching of audio is an interesting challenge for developers in the sense that audio could be searched by having the user draw the corresponding sound wave, but very few users can actually describe what a sound wave would look like. Another approach might be to have the user hum a few notes of the music to the computer via a microphone and have the search tool bring back the completed piece of music based on the notes that were hummed. Regardless of the approach, these are very interesting challenges for developers in term of searching audio based on the content of the sound.

There are many popular search engines today such as Yahoo, Lycos, Excite, HotBot, and AltaVista. These engines are primarily used for text retrieval purposes, but they do have some functionality regarding multimedia.

Yahoo provides an excellent medium for browsing images through its classification system. Yahoo also provides an Image Surfer feature, which allows users to search through images, but the feature still needs further development. "The Image Surfer, which is an earlier version of the software developed by Excalibur does not appear on the Yahoo home page at all. You must click on the ‘more’ option then scroll down to the very bottom of that page to find the link. The size of the collection in the image gallery that Yahoo offers is quite puny. There was not a single picture in it about Hawaii and there was only one image file for a `Van Gough’ search." (Jacso, 1) Aside from image searching, Yahoo also has very little functionality for searching audio. Yahoo classifies multimedia files very well within its well-known classification scheme, but the searching features for these types of files are lacking. They still require further development.

Lycos is further along than Yahoo as it provides a much larger gallery of images, which are classified into categories. Lycos also offers the ability to search these images as well as sound files, but they have to be done in a specific manner. "As the sound and image filters are radio buttons, they are mutually exclusive, so it is not possible to use them simultaneously to find sites with images or sound clips (let alone with both). You have to run two separate searches, one for sound and one for images." (Jacso, 1) Although Lycos does provide the user with ability to search images, it is not yet sophisticated enough to handle some of the more complex searches where images and audio are combined.

Excite provides functionality to search multimedia files based on their file formats. For example, Excite provides a feature in its "Advanced Search" template where the user can limit their search for only sites that contain video and/or audio files. This functionality is good if the user is specifically looking for information in a specific format, but it is not searching multimedia based on the content of those files. Also, Excite does not currently provide any easy way to search for images.

HotBot also has the ability to filter searches specifically for video, or audio files, and unlike the other search engines, it also has the ability to search for images. "HotBot is also the best in terms of results sets. For the rather specific search about Van Gough’s painting Chair with a Pipe, it returns 88 hits." (Jasco,1) HotBot has the ability to search for images, but not based on the content. The 88 hit result set for the Van Gough query was mostly likely text matches to the title, rather than search results based on the content of the painting.

AltaVista provides some features for image searching such as its Photo and Media Finder where the user can search for images. AltaVista also provides the most information in the results sets such as the file format, the file size, the color depth (for images), and the duration of audio and video files. In the case of image and video searches, "the results are immediately displayed in thumbnail formats, so you don’t waste time clicking on a site that mentions Van Gogh, but shows the picture of a fan of Van Gogh and his favorite cat instead of the artist’s work." This ability is vastly different from the other search engines, which merely provide listings.

Although many aspects of searching on the Web are still in the developmental phase, there is work being done to improve and expand searching, especially in the area of multimedia. As described above the most popular search engines today do provide some type of multimedia searching, but they still need work and additional development. Most of the popular search engines provide the ability to limit one’s search to only multimedia files (e.g., file extension of gif, jpg, wav), but this is not searching based on content. It simply allows users to only search multimedia files when they know that it is what they are looking for. Aside from the functionality provided by the search engines, the way in which the user searches the web also must change in order to make searching more efficient.

Currently, text searching is difficult because the user usually gets more information than he/she wants or not the right information In part, this is because most of the time, the user does not know what type of information they are looking for. In order to remedy this problem, in the article entitled, "Sorting Out Searching, A User-Interface Framework for Text Searches", Ben Shneiderman, Donald Byrd, and W. Bruce Croft propose a four-phase search framework consisting of "formulation (what happens before the user starts a search); action (starting the search); review of results (what the user sees resulting from the search); and refinement (what happens after review of results and before going back to formulation)." (Shneiderman, 96) This process is one that repeats as the user narrows down the search criteria to reach specifically what they are looking for.

The formulation step is when the user must make some very important decisions regarding their search such as what sources should they use in their search, what documents they should search, and what text to search for. The formulation step is typically the most complex as it requires the user to know have a good prior understanding of exactly what is the desired information. The action step is when the user actually begins to search. Typically, this is done by clicking the "Search" button for most search engines, but more and more "an appealing alternative is ‘dynamic queries’ in which there is no Search button, but the result set is continuously displayed and updated as the user changes the search. The dynamic-queries technique requires adequate screen space and rapid processing, but the advantages are great, allowing a user to broaden, narrow, and refocus a search several times in as many seconds."(Shneiderman, 97) The review results step is when the user analyses what the search tool has provided to determine if the information is what he/she desires or if additional searches need to be made. Finally refinement is the process where the user, having reviewed the results, determines how further searches can be changes, refined, and/or modified.

The four-phase framework model is an effort to standardize the searching process for the developers and the users benefit. Much of the idea described in the model could be provided by a search engine to some extent. Since Web is now viewed as a worldwide tool, "the future of the Web as a universal tool may depend on interface developers’ ability to reduce frustration and confusion for the masses of users while enabling them to reliably find what they need." (Shneiderman, 98)

Video searching provides some unique and interesting challenges for developers to come up with some sort of automated way of searching through video and/or still images. One method that is currently in development is to generate a storyboard out of a video. Storyboards typically consist "of a series of sketches showing each shot in each scene as it will be filmed, and possibly some indication of the action-taking place (e.g., an arrow showing the direction of movement. A ‘shot’ is defined as a section of action during which the camera films continuously without interruption." (Macer, 303) A storyboard is typically used by writers and directors while making a movie to plan the action of the shot, to review camera angles, and provide a summary of the film. Essentially, the proposed approach would be to take a finished video product and generate the storyboard based on the finish video. "In order to reverse-engineer a storyboard from the finished video sequence, it is necessary to identify three properties of each shot in the sequence. These are: (1) the start point of the shot, (2) the end point of the shot and (3) the picture that best represents the shot as a whole." (Macer,303) Once the storyboard has been generated, it will be easier to search for video sequences, especially in large video libraries. Essentially, the storyboard theory is to transform the video into still images in an effort to make search for specific video sequences and/or specific video clips easier.

Another approach to video searching is the search actual video (not frames or images) using video cues. At Columbia University, a system called VideoQ is being developed that does this. The theory behind this system is to have the user actually draw out an animated scene as the query. "In an animated sketch, motion and temporal duration are the key attributes assigned to each object in the sketch in addition to the usual attributes such as shape, color, and texture. Using the visual palette, we sketch out a scene by drawing a collection of video objects." (Chang, 314) The VideoQ system will then search its video library for videos that match the animated sketch. The VideoQ system is intended to be on the Internet and use various Java applets to allow the user to create these animated sketches.

In terms of interface design, it will be interesting to see what kind of interface is developed for the user to query using animated sketches. It would have to be an interface that provides the necessary functionality, but also is easy enough for the average user to comprehend. This would be especially important for this system as it is proposing a querying method that the average user would be unfamiliar with. The interface will be key in making the learning curve for these applications as smooth as possible, so that users become comfortable with the concepts proposed.

In terms of audio searching development, there is company based out of Grass Valley, California that is currently doing some developmental work in searching audio based on content rather than text description. This company is called Comparisonics and their new technology was debuted at the National Association of Broadcasters (NAB) convention in April of 1998. Essentially, the way the system works is by having the user initiate a sound-matching query either by using a sound synthesizer or by simply speaking into a microphone. Then, "the digital audio content is first characterized by an automated indexing process. Then any sound can be used to find matches in the indexed collection. The given sound, called a prototype, is compared automatically with each sound in the collection. The degree of similarity between two sounds is indicated by a similarity score that ranges from 0 to 100." (Rice, 1) The results with the highest similarity scores are displayed to the user. This technology is said to work for all types of sounds including people, animals, machinery, and musical instruments.

In terms of interfaces, Comparisonics has a very unique of displaying sounds to their users. All sound files are displayed in terms of their sound waves, but the different parts of the waves are color coded and labeled. For example, if one sound file contains multiple sounds such as a doorbell, footsteps, and a scream, each sound in the wave would be coded as a different color and labeled. The sound classification system can also be used for songs where each verse and the repeating chorus has its own color. This allows the user to actually "see" the sound and Comparisonics provides the user with additional functionality based on the color classification. The user can choose one sound in a multiple sound file and search the file for similar sounds if necessary. Also, the user could select a particular note in a song and search for the same note in other verses of the song. The Comparisonics search process is called content-based audio retrieval or sound matching. It will be interesting to see how the color-coding scheme is established. If the color scheme changes for every wave file, it may be more difficult for users to follow.

Aside from video, image and audio searching, geographical or place based searching is something that is also being developed, specifically for GIS systems. "The ability to search by place, or place-based search, is crucial for many applications of GIS. In emergency management, quickly finding all of the information relevant to a disaster such as an earthquake or a tornado is important, and relevance is determined in such cases primarily by geographic location." Geographical searching would involve being able to search about specific areas of the Earth’s surface based on geological and biological characteristics. This would be beneficial to scientists who want to conduct specific studies. Currently, place-base searching is limited to searching based on ZIP code. This is provided by the EPA web site. Although geographical searching is really only applicable to a specific user group, it is another aspect in which searching can expand for the future.

Searching has come a long way since the days of standing for hours at a library card catalog looking for books based on author or subject. With the advent of the Internet, information from all over the world is available to people from all over the world. Since there is so much information out there, people require an automated method to search through all of it. The search engines available today provide users with this ability, but primarily for text based searching. As the Internet moves further and further into being able to support multimedia, users and corporations will need to take advantage of new searching techniques. Some companies have even begun to hire "Web Specialists" to assist them in becoming aware of what searching facilities are out there and which engines work the best based on their needs.

The idea of being able to search images, video, or audio based on the content is possible since the Internet is an electronic medium, but something that is still in development. As more and more users begin to understand the concepts behind searching these media, the need to do so will rise. However, today, most of the work being done to allow users to search this media is still in the developmental phases. The approaches described above are simply ideas and theories for ways in which this type of searching could be possible. The idea of generating a storyboard from a video, or searching a video based on a moving sketch, or searching audio based on content and colored wave files are simply ways in which searching multimedia may become a reality.

In terms of interface designs, the ideas presented above bring new challenges to the developers to design interfaces that are simple enough for the average user to understand. It will take time for users to become comfortable with the idea of having to draw and assign movement to a query or to look at sounds based on color. The interface developed with these systems will have to be user friendly enough so that the average user will be able to reap the benefits of this type of searching.

Searching on the Web is a technology that is relatively young in terms of multimedia. Text searching has been around the longest, but even it has some ways to improve. As video and audio become more of an electronic medium for communication, the possibilities for searching these media are endless. It will be interesting to watch this area of technology evolve in the years to come.

Chang, Shih-Fu, William Chen, Horace J. Meng, Hari Sundaram, and Di Zhong "ViedoQ: An Automated Content Based Video Search System Using Visual Cues", Communications of the ACM, 1997

Ellsworth, Jill "Time To Consider a Web Search Specialist" Working The Net, September, 1999

Ikeji, Augustine Chidi, and Farshad Fotouhi "An Adaptive Real-Time Web Search Engine" Communications of the ACM, 1999

Jasco, Peter "Sorting Out the Wheat from the Chaff: Take a Look at the Multimedia Features of Seven Web-Wide Search Services" MIS Quarterly

Kaindl, Herman, Stephen Kramer, Luis Miguel Afonso "Combining Structure Search and Content Search for the World-Wide Web" Communications of the ACM, 1998

Macer, Peter, Peter J. Thomas, Nouhman Chalabi, John F. Meech "Finding the Cut of the Wrong Trousers: Fast Video Search Using Automatic Storyboard Generation" Communications of the ACM, 1996

Rice, Stephen V. "The Comparisonics White Paper" www.comparisonics.com April 1998

Saito, Mari, Kazunori Ohmura "A Cognitive Model for Searching for Ill-defined Targets on the Web" Communications of the ACM, 1998

Shneirderman, Ben Designing the User Interface: Strategies for Effective Human – Computer Interaction, Addison – Wesley, Copyright, 1998

Shneiderman, Ben, Donald Byrd, and W. Bruce Croft "Sorting Out Searching: A User-Interface Framework for Text Searches" Communications of the ACM, 1998

Tosta, Nancy "Eyes Wide Open" Geo Info Systems Showcase, August 1998

Turoff, Murray CIS 447/732 Human Computer Interface Draft Book, Copyright, 1999