{"id":57,"date":"2018-07-05T11:23:13","date_gmt":"2018-07-05T11:23:13","guid":{"rendered":"http:\/\/lisp7.epgpbooks.inflibnet.ac.in\/?post_type=chapter&p=57"},"modified":"2018-08-01T05:31:47","modified_gmt":"2018-08-01T05:31:47","slug":"multimedia-information-retrieval","status":"publish","type":"chapter","link":"https:\/\/ebooks.inflibnet.ac.in\/lisp7\/chapter\/multimedia-information-retrieval\/","title":{"rendered":"Multimedia Information Retrieval"},"content":{"raw":"
\r\n\r\n \r\n\r\nI.\u00a0 <\/strong>Objectives<\/strong>\r\n\r\n \r\n\r\n \r\n\r\nThe objective of this module is to:\r\n\r\n \r\n\r\n\u2022\u00a0\u00a0\u00a0 Describe the basic concept of Multimedia Information system.\r\n\r\n\u2022\u00a0\u00a0\u00a0 Introduce the reader about various scenarios where non-textual or multimedia information plays major role.\r\n\r\n\u2022\u00a0\u00a0\u00a0 Familiarize the reader with the various functions of Multimedia Information Retrieval.\r\n\r\n\u2022\u00a0\u00a0\u00a0 Enlist different types of Multimedia Information Retrieval.\r\n\r\n\u2022\u00a0\u00a0\u00a0 Introduce various approaches to handle different Multimedia Information Retrieval systems.\r\n\r\n \r\n\r\n \r\n\r\n \r\n\r\nII.\u00a0\u00a0 Learning Outcomes\u00a0<\/strong>\r\n\r\n \r\n\r\nAfter reading this module:\r\n\r\n \r\n\r\n\u2022\u00a0\u00a0\u00a0 The reader will gain the knowledge of different multimedia objects.\r\n\r\n\u2022\u00a0\u00a0\u00a0 The reader will gain a good understanding of Multimedia Information Retrieval System.\r\n\r\n\u2022\u00a0\u00a0\u00a0 The reader will be know about the different subfields of Multimedia Information Retrieval.\r\n\r\n\u2022\u00a0\u00a0\u00a0 The reader will gain the knowledge of various information retrieval techniques for retrieval of multimedia information like Audio, Video, etc.\r\n\r\n\u2022\u00a0\u00a0\u00a0 The \u00a0reader \u00a0will \u00a0understand \u00a0the \u00a0various \u00a0research \u00a0areas \u00a0in \u00a0Multimedia \u00a0Information Retrieval.\r\n\r\n \r\n\r\n \r\n\r\nIII.\u00a0\u00a0 Structure\u00a0<\/strong>\r\n\r\n \r\n\r\n1.\u00a0 Introduction\r\n\r\n2.\u00a0 Multimedia Information Retrieval\r\n\r\n3.\u00a0 Text Information Retrieval\r\n\r\n4.\u00a0 Audio and Music Information Retrieval\r\n\r\n5.\u00a0 Image Information Retrieval\r\n\r\n6.\u00a0 Video Information Retrieval\r\n\r\n7.\u00a0 Summary\r\n\r\n8.\u00a0 References\r\n\r\n<\/div>\r\n \r\n
\r\n\r\n \r\n\r\n1.\u00a0 Introduction\u00a0<\/strong>\r\n\r\n \r\n

Text has been the predominant medium for the communication of information. With the availability of better computing capabilities such as availability of disk space, memory space, display resolution and better processing power the other media such as image, audio and video etc. are also gaining importance. There are many fields of work that require access to non-textual information. For example, medical professionals need access to medical images, architects to building plans, ornithologists to bird calls, estate agents to property photographs, car engineers and buyers need photographs and sound of car engines, and so on. This gave rise to another kind of systems that have been developed to handle information contained in more than one medium, known as multimedia information systems.<\/p>\r\n \r\n

In the other sections of this course, we are already aware that the text information retrieval has been already established well. However, multimedia information retrieval is less established. For having good understanding about the multimedia information retrieval it is very necessary to understand the basics of text retrieval, audio and music information retrieval, image and video information retrieval.<\/p>\r\n \r\n\r\nIn this module we will discuss the definition, approaches and applications of each one of the above.\r\n\r\n \r\n\r\n2.\u00a0 Multimedia Information Retrieval\u00a0<\/strong>\r\n\r\n \r\n

Multimedia information retrieval system is associated with storage, indexing, search, and delivery of multimedia data such as images, videos, sounds, 3D graphics, or their combination. By definition, it includes works on, for example, extracting descriptive features from images, reducing high dimensional indexes into low-dimensional ones, defining new similarity metrics, efficient delivery of the retrieved data, and so forth. Systems that provide all or part of the above functionalities are multimedia retrieval systems. The Google image search engine is a typical example of such a system. A video-on-demand site that allows people to search movies by their titles is another example.<\/p>\r\n \r\n

Multimedia information has some specific characteristics that make it distinct from textual information; thus multimedia information retrieval systems differ from conventional text retrieval systems. A good multimedia information retrieval system should have the capability to store, retrieve and present heterogeneous data ranging from text to audio, still and moving images and digital video. The architecture of a multimedia information retrieval system depends on the characteristics of the multimedia data and the kind of operations to be performed on such data.<\/p>\r\n \r\n\r\nMultimedia information retrieval encompasses different subareas:\r\n\r\n \r\n\r\n\u2022\u00a0\u00a0\u00a0 Content representation and multimedia object representation.\r\n\r\n\u2022\u00a0\u00a0\u00a0 Feature extraction\r\n\r\n\u2022\u00a0\u00a0\u00a0 Query formulation to map high-level semantic concepts into low level features\r\n\r\n\u2022\u00a0\u00a0\u00a0 Query-by-example\r\n\r\n\u2022\u00a0\u00a0\u00a0 Relevance feedback and interactive queries<\/span>\r\n\r\n\u2022\u00a0\u00a0\u00a0 Efficient feature indexing and cataloguing<\/span>\r\n\r\n\u2022\u00a0\u00a0\u00a0 Integrated searching and browsing<\/span>\r\n\r\n\u2022\u00a0\u00a0\u00a0 Techniques of searching multimedia based on their contents<\/span>\r\n\r\n<\/div>\r\n

\r\n\r\n \r\n

In short we can say, multimedia information retrieval is retrieval of text, image, video, and sound data related to the user and their ranking according to some similarity degree. The better the similarity degree there will be more likelihood of user finding the relevant answers.<\/p>\r\n \r\n\r\n3.\u00a0 Text Information Retrieval\u00a0<\/strong>\r\n\r\n \r\n

Text information retrieval is now very well established and developed. It basically involves answering user queries based on a keyword index. In a typical session, a user frames his query in the form of some keywords in a search bar. The keywords received through the search bar are processed using various techniques of spell check, tokenization and applying logical operators. The processed query is matched with the already created index of documents present in the information system. The result of this matching is displayed to the user specifying the location of documents. Finally the user selects relevant document and if dissatisfied, reframes the query.<\/p>\r\n \r\n\r\nIn other modules text information retrieval has been discussed in detail.\r\n\r\n \r\n\r\n4.\u00a0 Audio and Music Information Retrieval\u00a0<\/strong>\r\n\r\n \r\n

Audio and music information retrieval have become prominent areas of research over the past few years. The most easy and commonly used medium of communication is speech. Although tools have been developed for capturing and storing the audio information, due to its sequential nature it becomes very challenging to retrieve a particular piece of audio from a long recorded audio. Another problem is that of losing the context of the retrieved audio.<\/p>\r\n \r\n

The conventional text retrieval techniques may be applied to voice retrieval easily if we could generate the transcripts of the spoken audio documents. Advances in speech recognition have made it possible to automatically generate good quality speech transcripts. A perfect automatic speech recognition (ASR) system that can efficiently transcribe spoken audio document would be an ideal solution. Hidden Markov Models (HMM) form the backbone of ASR systems. An HMM is a statistical representation of a speech event like a word. Model parameters are trained on a large corpus of labelled speech data. Once a trained set of HMMs is generated, query speech can be matched to find the most likely model sequence (the recognized words). However, even good quality transcript lacks punctuation, paragraphs, and all the elements that provide structure. Although the retrieval based on speech transcripts seems to be very close to text retrieval, in practice it is not so. Out-of-vocabulary words, such as proper nouns cannot be recognized by many ASR systems. Another problem is that the whole process is very time consuming and expensive as the audio documents usually have large sized file.<\/p>\r\n \r\n

Similarly, music information retrieval is also another area of research in multimedia information retrieval. This is very less developed field as it does not contain any specific words per se. Transcript generation in not possible in musical information. Music information consists of seven facets:<\/p>\r\n\r\n<\/div>\r\n

\r\n\r\n \r\n\r\n\u2022\u00a0\u00a0\u00a0 Pitch: a quality of sound that is related to the frequency\r\n\r\n\u2022\u00a0\u00a0\u00a0 Tempo: information concerning the duration of a musical event\r\n\r\n\u2022\u00a0\u00a0\u00a0 Harmony: related to the attribute of music\r\n\r\n\u2022\u00a0\u00a0\u00a0 Timbre: related to tone\r\n\r\n\u2022\u00a0\u00a0\u00a0 Editing: related to the performance instructions as fingering, ornamentation, articulations and so on.\r\n\r\n\u2022\u00a0\u00a0\u00a0 Text: related to the lyrics, symphonies and so on.\r\n\r\n\u2022\u00a0\u00a0\u00a0 Bibliography: information about the composer, performer, title of the piece, publisher etc.\r\n\r\n \r\n

A query based music retrieval system relies on similarity matching between the query and the stored music. The user is provided with an interface where the user submits query by playing the music or humming a tune. Then the received query is transformed into digital format and matched with the available recordings in the database and the most relevant strings are returned to the user. Some other advanced approaches involve matching the different attributes discussed above, such as comparing the edit distances, matching the pitch contours, matching the time contours that represent the rhythm information.<\/p>\r\n \r\n

In other approaches, text based retrieval techniques are used for music retrieval such as, providing search by the name of the artist, title of the song, file types, popularity ratings and by keywords.<\/p>\r\n \r\n

Audio retrieval research faces many challenges because of two distinct qualities of audio data: audio data is aurally based instead of visually based, and audio data is time-dependent. Also the presentation of retrieved results is also a challenge, as the audio data is time-dependent a user when supplied with 20 clips and simultaneously played would be of no use to the searcher. There are some efforts going on for finding how to browse and navigate through databases of audio.<\/p>\r\n \r\n\r\n5.\u00a0 Image Information Retrieval\u00a0<\/strong>\r\n\r\n \r\n

Among audio, video and image, image information retrieval is the best developed technology, since even before modern civilization; images are being used as a major medium for communication. With over a decade of research and development, image information retrieval has had time to grow and mature. However, image processing and retrieval activities began in the 1980s, and became active area of research interest since the creation of the web in the early 1990s.<\/p>\r\n \r\n

Several commercial image data management systems provide retrieval based on metadata and text keywords or assigned descriptors. These metadata and descriptors are manually provided by the human indexers by describing the various attributes of the image. The process of describing images by human indexers is very expensive and time-consuming process, and yet is highly subjective. The attributes that can be used for retrieval are:<\/p>\r\n \r\n\r\n\u2022\u00a0\u00a0\u00a0 A combination of colour, texture, shape and so on\r\n\r\n\u2022\u00a0\u00a0\u00a0 A specific arrangement of objects in the image\r\n\r\n\u2022\u00a0\u00a0\u00a0 Depiction of a particular event\r\n\r\n\u2022\u00a0\u00a0\u00a0 Presence of one or more persons or objects<\/span>\r\n\r\n\u2022\u00a0\u00a0\u00a0 Presence of a specific location<\/span>\r\n\r\n\u2022\u00a0\u00a0\u00a0 Emotions attached to an event or a person, and so on.<\/span>\r\n\r\n<\/div>\r\n

\r\n\r\n \r\n\r\nAll these attributes are researched under the category of \u201ccontext based image retrieval (CBIR)\u201d Two different types of interfaces are used for querying images:\r\n\r\n \r\n\r\ni.\u00a0\u00a0<\/strong>Browsing and navigation interface<\/strong>: here the user is allowed to browse a collection and navigate through a structured collection to images in order of retrieve the desired images(s).\r\n\r\n \r\n

ii.\u00a0<\/strong>Query interface<\/strong>: in most cases a query by example approach is followed whereby the user can specify an image from a collection, which is used as a query to search the database. The problem of this approach is that the system should have a selection of sample objects, with associated attributes, which can be used for querying the database. Some interfaces offer options for selection from a palette or sketch input.<\/p>\r\n \r\n\r\nSimilarity between a query image and the image objects in the collection is computed and images that fully or partially match the query are retrieved.\r\n\r\n \r\n

Other approaches involve, answering the query based on colour histogram. Colour histogram shows the proportion of pixels of each colour within the image. A colour histogram is computed for each image in the database and stored internally as vector of values which are easier for matching algorithms to match with the query. This approach solves queries like \u201cfind all images whose most frequently used colour is similar to this image\u201d.<\/p>\r\n \r\n\r\n6.\u00a0 Video Information Retrieval\u00a0<\/strong>\r\n\r\n \r\n

Video data retrieval shares some properties with image data retrieval, due to the commonality of their visual nature. However, the video information is time-dependent like audio information, which means the video information changes as the time changes; also usually video is associated with synchronized audio track, the image and audio retrieval techniques are applied on video retrieval too. Videos are usually made up of a number of distinct scenes each of which can be further broken down to individual shots depicting a single view, conversation or action.<\/p>\r\n \r\n

Video retrieval systems are still in early stage of research and development. There are some popular approaches being followed in video retrieval. Application of text retrieval techniques combined with content-based image retrieval techniques (CBIR) is deemed as the best solution. Another one is by indexing of video metadata, colour histograms, texture analysis, video segmentation (breaking the video into small segments, usually where the camera shot changes), pattern recognition.<\/p>\r\n \r\n

In another approach, if closed-captioning signals exists for a video, the keywords are extracted from the text of the closed captioning, using well understood text manipulation techniques. If the closed-captioning does not exist the keywords are extracted from the audio stream.<\/p>\r\n\r\n<\/div>\r\n

Another strategy involves use of key frames. Key frames are frames whose images represent a semantic unit of the stream, such as a scene. After extracting the key frames, image retrieval techniques could be applied to support queries on key frames.<\/span><\/p>\r\n\r\n

\r\n\r\n \r\n\r\n7.\u00a0 Summary\u00a0<\/strong>\r\n\r\n \r\n\r\nIn this Module we tried to understand the answers to the following questions:\r\n\r\n \r\n\r\n\u2022\u00a0\u00a0\u00a0 What is multimedia information retrieval?\r\n\r\n\u2022\u00a0\u00a0\u00a0 What is text, image, audio and video information retrieval?\r\n\r\n\u2022\u00a0\u00a0\u00a0 What are the various approaches being followed for the retrieval of the above?\r\n\r\n\u2022\u00a0\u00a0\u00a0 What are the areas of application of multimedia information retrieval?\r\n\r\n \r\n

Multimedia offers a richer experience than plain text as they have other details which were otherwise not possible to express in text only format. We understood the various aspects and properties of image, text, audio, music, and video information representations and the concept of multimedia information retrieval and types of media that forms multimedia. Though the text is most prevalent format for information the underlying technology and developments in infrastructure are making other forms more visible. The approaches being used for developing multimedia information retrieval systems are also discussed.<\/p>\r\n \r\n\r\n \r\n\r\n8.\u00a0 References\u00a0<\/strong>\r\n\r\n \r\n\r\n1. Chaudhary, G.G. Introduction to modern information retrieval, 2nd \u00a0ed. London, Facet Publishing, 2004\r\n\r\n2. Downie, J. S., Music Information retrieval. In Cronin, B. (ed.) ARIST, 37, Information Today, 2002.\r\n\r\n3. Eakins, J. P. and Graham, M.E., Context-based image retrieval: a report to the JISC technology application programme, 1999.\r\n\r\n4. Eakins, J. P., Techniques for image retrieval, Library and Information Briefings, 85, south bank University, 1998.\r\n\r\n5. Faloutsos, C., Multimedia IR: indexing and searching. In Baeza-, R. and Ribeiro-Neto, B., Modern information retrieval, ACM Press, 1999.\r\n\r\n6. Gudivada,\u00a0 V. \u00a0N. \u00a0and \u00a0Raghavan, \u00a0V. V., \u00a0Modeling \u00a0and \u00a0retrieving \u00a0images \u00a0by\u00a0 content.\r\n\r\nInformation processing & management, 33(4), 427-452, 1997.\r\n\r\n7. Khosrow-Pour, Mehdi(ed.), Dictionary of information science and technology, Hershey, Idea Group Reference, 2006\r\n\r\n8. Lancaster, F.W., Indexing and abstracting theory and practice, 3rd ed. London, Facet Publishing, 2003.<\/span>\r\n\r\n9. Rasmussen, E., Libraries and bibliographic systems. In Baeza-Yates, R. and Ribeiro- Neto, B., Modern information retrieval, ACM Press, 1999.<\/span>\r\n\r\n10. Tang, Nelson, \u00a0and Jonathan Furner. \u00a0\"Multimedia \u00a0Information\u00a0 Retrieval \u00a0Systems: An Overview\", 1999.<\/span>\r\n\r\n<\/div>","rendered":"

\n

 <\/p>\n

I.\u00a0 <\/strong>Objectives<\/strong><\/p>\n

 <\/p>\n

 <\/p>\n

The objective of this module is to:<\/p>\n

 <\/p>\n

\u2022\u00a0\u00a0\u00a0 Describe the basic concept of Multimedia Information system.<\/p>\n

\u2022\u00a0\u00a0\u00a0 Introduce the reader about various scenarios where non-textual or multimedia information plays major role.<\/p>\n

\u2022\u00a0\u00a0\u00a0 Familiarize the reader with the various functions of Multimedia Information Retrieval.<\/p>\n

\u2022\u00a0\u00a0\u00a0 Enlist different types of Multimedia Information Retrieval.<\/p>\n

\u2022\u00a0\u00a0\u00a0 Introduce various approaches to handle different Multimedia Information Retrieval systems.<\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

II.\u00a0\u00a0 Learning Outcomes\u00a0<\/strong><\/p>\n

 <\/p>\n

After reading this module:<\/p>\n

 <\/p>\n

\u2022\u00a0\u00a0\u00a0 The reader will gain the knowledge of different multimedia objects.<\/p>\n

\u2022\u00a0\u00a0\u00a0 The reader will gain a good understanding of Multimedia Information Retrieval System.<\/p>\n

\u2022\u00a0\u00a0\u00a0 The reader will be know about the different subfields of Multimedia Information Retrieval.<\/p>\n

\u2022\u00a0\u00a0\u00a0 The reader will gain the knowledge of various information retrieval techniques for retrieval of multimedia information like Audio, Video, etc.<\/p>\n

\u2022\u00a0\u00a0\u00a0 The \u00a0reader \u00a0will \u00a0understand \u00a0the \u00a0various \u00a0research \u00a0areas \u00a0in \u00a0Multimedia \u00a0Information Retrieval.<\/p>\n

 <\/p>\n

 <\/p>\n

III.\u00a0\u00a0 Structure\u00a0<\/strong><\/p>\n

 <\/p>\n

1.\u00a0 Introduction<\/p>\n

2.\u00a0 Multimedia Information Retrieval<\/p>\n

3.\u00a0 Text Information Retrieval<\/p>\n

4.\u00a0 Audio and Music Information Retrieval<\/p>\n

5.\u00a0 Image Information Retrieval<\/p>\n

6.\u00a0 Video Information Retrieval<\/p>\n

7.\u00a0 Summary<\/p>\n

8.\u00a0 References<\/p>\n<\/div>\n

 <\/p>\n

\n

 <\/p>\n

1.\u00a0 Introduction\u00a0<\/strong><\/p>\n

 <\/p>\n

Text has been the predominant medium for the communication of information. With the availability of better computing capabilities such as availability of disk space, memory space, display resolution and better processing power the other media such as image, audio and video etc. are also gaining importance. There are many fields of work that require access to non-textual information. For example, medical professionals need access to medical images, architects to building plans, ornithologists to bird calls, estate agents to property photographs, car engineers and buyers need photographs and sound of car engines, and so on. This gave rise to another kind of systems that have been developed to handle information contained in more than one medium, known as multimedia information systems.<\/p>\n

 <\/p>\n

In the other sections of this course, we are already aware that the text information retrieval has been already established well. However, multimedia information retrieval is less established. For having good understanding about the multimedia information retrieval it is very necessary to understand the basics of text retrieval, audio and music information retrieval, image and video information retrieval.<\/p>\n

 <\/p>\n

In this module we will discuss the definition, approaches and applications of each one of the above.<\/p>\n

 <\/p>\n

2.\u00a0 Multimedia Information Retrieval\u00a0<\/strong><\/p>\n

 <\/p>\n

Multimedia information retrieval system is associated with storage, indexing, search, and delivery of multimedia data such as images, videos, sounds, 3D graphics, or their combination. By definition, it includes works on, for example, extracting descriptive features from images, reducing high dimensional indexes into low-dimensional ones, defining new similarity metrics, efficient delivery of the retrieved data, and so forth. Systems that provide all or part of the above functionalities are multimedia retrieval systems. The Google image search engine is a typical example of such a system. A video-on-demand site that allows people to search movies by their titles is another example.<\/p>\n

 <\/p>\n

Multimedia information has some specific characteristics that make it distinct from textual information; thus multimedia information retrieval systems differ from conventional text retrieval systems. A good multimedia information retrieval system should have the capability to store, retrieve and present heterogeneous data ranging from text to audio, still and moving images and digital video. The architecture of a multimedia information retrieval system depends on the characteristics of the multimedia data and the kind of operations to be performed on such data.<\/p>\n

 <\/p>\n

Multimedia information retrieval encompasses different subareas:<\/p>\n

 <\/p>\n

\u2022\u00a0\u00a0\u00a0 Content representation and multimedia object representation.<\/p>\n

\u2022\u00a0\u00a0\u00a0 Feature extraction<\/p>\n

\u2022\u00a0\u00a0\u00a0 Query formulation to map high-level semantic concepts into low level features<\/p>\n

\u2022\u00a0\u00a0\u00a0 Query-by-example<\/p>\n

\u2022\u00a0\u00a0\u00a0 Relevance feedback and interactive queries<\/span><\/p>\n

\u2022\u00a0\u00a0\u00a0 Efficient feature indexing and cataloguing<\/span><\/p>\n

\u2022\u00a0\u00a0\u00a0 Integrated searching and browsing<\/span><\/p>\n

\u2022\u00a0\u00a0\u00a0 Techniques of searching multimedia based on their contents<\/span><\/p>\n<\/div>\n

\n

 <\/p>\n

In short we can say, multimedia information retrieval is retrieval of text, image, video, and sound data related to the user and their ranking according to some similarity degree. The better the similarity degree there will be more likelihood of user finding the relevant answers.<\/p>\n

 <\/p>\n

3.\u00a0 Text Information Retrieval\u00a0<\/strong><\/p>\n

 <\/p>\n

Text information retrieval is now very well established and developed. It basically involves answering user queries based on a keyword index. In a typical session, a user frames his query in the form of some keywords in a search bar. The keywords received through the search bar are processed using various techniques of spell check, tokenization and applying logical operators. The processed query is matched with the already created index of documents present in the information system. The result of this matching is displayed to the user specifying the location of documents. Finally the user selects relevant document and if dissatisfied, reframes the query.<\/p>\n

 <\/p>\n

In other modules text information retrieval has been discussed in detail.<\/p>\n

 <\/p>\n

4.\u00a0 Audio and Music Information Retrieval\u00a0<\/strong><\/p>\n

 <\/p>\n

Audio and music information retrieval have become prominent areas of research over the past few years. The most easy and commonly used medium of communication is speech. Although tools have been developed for capturing and storing the audio information, due to its sequential nature it becomes very challenging to retrieve a particular piece of audio from a long recorded audio. Another problem is that of losing the context of the retrieved audio.<\/p>\n

 <\/p>\n

The conventional text retrieval techniques may be applied to voice retrieval easily if we could generate the transcripts of the spoken audio documents. Advances in speech recognition have made it possible to automatically generate good quality speech transcripts. A perfect automatic speech recognition (ASR) system that can efficiently transcribe spoken audio document would be an ideal solution. Hidden Markov Models (HMM) form the backbone of ASR systems. An HMM is a statistical representation of a speech event like a word. Model parameters are trained on a large corpus of labelled speech data. Once a trained set of HMMs is generated, query speech can be matched to find the most likely model sequence (the recognized words). However, even good quality transcript lacks punctuation, paragraphs, and all the elements that provide structure. Although the retrieval based on speech transcripts seems to be very close to text retrieval, in practice it is not so. Out-of-vocabulary words, such as proper nouns cannot be recognized by many ASR systems. Another problem is that the whole process is very time consuming and expensive as the audio documents usually have large sized file.<\/p>\n

 <\/p>\n

Similarly, music information retrieval is also another area of research in multimedia information retrieval. This is very less developed field as it does not contain any specific words per se. Transcript generation in not possible in musical information. Music information consists of seven facets:<\/p>\n<\/div>\n

\n

 <\/p>\n

\u2022\u00a0\u00a0\u00a0 Pitch: a quality of sound that is related to the frequency<\/p>\n

\u2022\u00a0\u00a0\u00a0 Tempo: information concerning the duration of a musical event<\/p>\n

\u2022\u00a0\u00a0\u00a0 Harmony: related to the attribute of music<\/p>\n

\u2022\u00a0\u00a0\u00a0 Timbre: related to tone<\/p>\n

\u2022\u00a0\u00a0\u00a0 Editing: related to the performance instructions as fingering, ornamentation, articulations and so on.<\/p>\n

\u2022\u00a0\u00a0\u00a0 Text: related to the lyrics, symphonies and so on.<\/p>\n

\u2022\u00a0\u00a0\u00a0 Bibliography: information about the composer, performer, title of the piece, publisher etc.<\/p>\n

 <\/p>\n

A query based music retrieval system relies on similarity matching between the query and the stored music. The user is provided with an interface where the user submits query by playing the music or humming a tune. Then the received query is transformed into digital format and matched with the available recordings in the database and the most relevant strings are returned to the user. Some other advanced approaches involve matching the different attributes discussed above, such as comparing the edit distances, matching the pitch contours, matching the time contours that represent the rhythm information.<\/p>\n

 <\/p>\n

In other approaches, text based retrieval techniques are used for music retrieval such as, providing search by the name of the artist, title of the song, file types, popularity ratings and by keywords.<\/p>\n

 <\/p>\n

Audio retrieval research faces many challenges because of two distinct qualities of audio data: audio data is aurally based instead of visually based, and audio data is time-dependent. Also the presentation of retrieved results is also a challenge, as the audio data is time-dependent a user when supplied with 20 clips and simultaneously played would be of no use to the searcher. There are some efforts going on for finding how to browse and navigate through databases of audio.<\/p>\n

 <\/p>\n

5.\u00a0 Image Information Retrieval\u00a0<\/strong><\/p>\n

 <\/p>\n

Among audio, video and image, image information retrieval is the best developed technology, since even before modern civilization; images are being used as a major medium for communication. With over a decade of research and development, image information retrieval has had time to grow and mature. However, image processing and retrieval activities began in the 1980s, and became active area of research interest since the creation of the web in the early 1990s.<\/p>\n

 <\/p>\n

Several commercial image data management systems provide retrieval based on metadata and text keywords or assigned descriptors. These metadata and descriptors are manually provided by the human indexers by describing the various attributes of the image. The process of describing images by human indexers is very expensive and time-consuming process, and yet is highly subjective. The attributes that can be used for retrieval are:<\/p>\n

 <\/p>\n

\u2022\u00a0\u00a0\u00a0 A combination of colour, texture, shape and so on<\/p>\n

\u2022\u00a0\u00a0\u00a0 A specific arrangement of objects in the image<\/p>\n

\u2022\u00a0\u00a0\u00a0 Depiction of a particular event<\/p>\n

\u2022\u00a0\u00a0\u00a0 Presence of one or more persons or objects<\/span><\/p>\n

\u2022\u00a0\u00a0\u00a0 Presence of a specific location<\/span><\/p>\n

\u2022\u00a0\u00a0\u00a0 Emotions attached to an event or a person, and so on.<\/span><\/p>\n<\/div>\n

\n

 <\/p>\n

All these attributes are researched under the category of \u201ccontext based image retrieval (CBIR)\u201d Two different types of interfaces are used for querying images:<\/p>\n

 <\/p>\n

i.\u00a0\u00a0<\/strong>Browsing and navigation interface<\/strong>: here the user is allowed to browse a collection and navigate through a structured collection to images in order of retrieve the desired images(s).<\/p>\n

 <\/p>\n

ii.\u00a0<\/strong>Query interface<\/strong>: in most cases a query by example approach is followed whereby the user can specify an image from a collection, which is used as a query to search the database. The problem of this approach is that the system should have a selection of sample objects, with associated attributes, which can be used for querying the database. Some interfaces offer options for selection from a palette or sketch input.<\/p>\n

 <\/p>\n

Similarity between a query image and the image objects in the collection is computed and images that fully or partially match the query are retrieved.<\/p>\n

 <\/p>\n

Other approaches involve, answering the query based on colour histogram. Colour histogram shows the proportion of pixels of each colour within the image. A colour histogram is computed for each image in the database and stored internally as vector of values which are easier for matching algorithms to match with the query. This approach solves queries like \u201cfind all images whose most frequently used colour is similar to this image\u201d.<\/p>\n

 <\/p>\n

6.\u00a0 Video Information Retrieval\u00a0<\/strong><\/p>\n

 <\/p>\n

Video data retrieval shares some properties with image data retrieval, due to the commonality of their visual nature. However, the video information is time-dependent like audio information, which means the video information changes as the time changes; also usually video is associated with synchronized audio track, the image and audio retrieval techniques are applied on video retrieval too. Videos are usually made up of a number of distinct scenes each of which can be further broken down to individual shots depicting a single view, conversation or action.<\/p>\n

 <\/p>\n

Video retrieval systems are still in early stage of research and development. There are some popular approaches being followed in video retrieval. Application of text retrieval techniques combined with content-based image retrieval techniques (CBIR) is deemed as the best solution. Another one is by indexing of video metadata, colour histograms, texture analysis, video segmentation (breaking the video into small segments, usually where the camera shot changes), pattern recognition.<\/p>\n

 <\/p>\n

In another approach, if closed-captioning signals exists for a video, the keywords are extracted from the text of the closed captioning, using well understood text manipulation techniques. If the closed-captioning does not exist the keywords are extracted from the audio stream.<\/p>\n<\/div>\n

Another strategy involves use of key frames. Key frames are frames whose images represent a semantic unit of the stream, such as a scene. After extracting the key frames, image retrieval techniques could be applied to support queries on key frames.<\/span><\/p>\n

\n

 <\/p>\n

7.\u00a0 Summary\u00a0<\/strong><\/p>\n

 <\/p>\n

In this Module we tried to understand the answers to the following questions:<\/p>\n

 <\/p>\n

\u2022\u00a0\u00a0\u00a0 What is multimedia information retrieval?<\/p>\n

\u2022\u00a0\u00a0\u00a0 What is text, image, audio and video information retrieval?<\/p>\n

\u2022\u00a0\u00a0\u00a0 What are the various approaches being followed for the retrieval of the above?<\/p>\n

\u2022\u00a0\u00a0\u00a0 What are the areas of application of multimedia information retrieval?<\/p>\n

 <\/p>\n

Multimedia offers a richer experience than plain text as they have other details which were otherwise not possible to express in text only format. We understood the various aspects and properties of image, text, audio, music, and video information representations and the concept of multimedia information retrieval and types of media that forms multimedia. Though the text is most prevalent format for information the underlying technology and developments in infrastructure are making other forms more visible. The approaches being used for developing multimedia information retrieval systems are also discussed.<\/p>\n

 <\/p>\n

 <\/p>\n

8.\u00a0 References\u00a0<\/strong><\/p>\n

 <\/p>\n

1. Chaudhary, G.G. Introduction to modern information retrieval, 2nd \u00a0ed. London, Facet Publishing, 2004<\/p>\n

2. Downie, J. S., Music Information retrieval. In Cronin, B. (ed.) ARIST, 37, Information Today, 2002.<\/p>\n

3. Eakins, J. P. and Graham, M.E., Context-based image retrieval: a report to the JISC technology application programme, 1999.<\/p>\n

4. Eakins, J. P., Techniques for image retrieval, Library and Information Briefings, 85, south bank University, 1998.<\/p>\n

5. Faloutsos, C., Multimedia IR: indexing and searching. In Baeza-, R. and Ribeiro-Neto, B., Modern information retrieval, ACM Press, 1999.<\/p>\n

6. Gudivada,\u00a0 V. \u00a0N. \u00a0and \u00a0Raghavan, \u00a0V. V., \u00a0Modeling \u00a0and \u00a0retrieving \u00a0images \u00a0by\u00a0 content.<\/p>\n

Information processing & management, 33(4), 427-452, 1997.<\/p>\n

7. Khosrow-Pour, Mehdi(ed.), Dictionary of information science and technology, Hershey, Idea Group Reference, 2006<\/p>\n

8. Lancaster, F.W., Indexing and abstracting theory and practice, 3rd ed. London, Facet Publishing, 2003.<\/span><\/p>\n

9. Rasmussen, E., Libraries and bibliographic systems. In Baeza-Yates, R. and Ribeiro- Neto, B., Modern information retrieval, ACM Press, 1999.<\/span><\/p>\n

10. Tang, Nelson, \u00a0and Jonathan Furner. \u00a0“Multimedia \u00a0Information\u00a0 Retrieval \u00a0Systems: An Overview”, 1999.<\/span><\/p>\n<\/div>\n","protected":false},"author":4,"menu_order":8,"template":"","meta":{"_acf_changed":false,"pb_show_title":"on","pb_short_title":"","pb_subtitle":"","pb_authors":["dr-vinit-kumar"],"pb_section_license":""},"chapter-type":[],"contributor":[63],"license":[],"class_list":["post-57","chapter","type-chapter","status-publish","hentry","contributor-dr-vinit-kumar"],"part":3,"_links":{"self":[{"href":"https:\/\/ebooks.inflibnet.ac.in\/lisp7\/wp-json\/pressbooks\/v2\/chapters\/57","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ebooks.inflibnet.ac.in\/lisp7\/wp-json\/pressbooks\/v2\/chapters"}],"about":[{"href":"https:\/\/ebooks.inflibnet.ac.in\/lisp7\/wp-json\/wp\/v2\/types\/chapter"}],"author":[{"embeddable":true,"href":"https:\/\/ebooks.inflibnet.ac.in\/lisp7\/wp-json\/wp\/v2\/users\/4"}],"version-history":[{"count":4,"href":"https:\/\/ebooks.inflibnet.ac.in\/lisp7\/wp-json\/pressbooks\/v2\/chapters\/57\/revisions"}],"predecessor-version":[{"id":120,"href":"https:\/\/ebooks.inflibnet.ac.in\/lisp7\/wp-json\/pressbooks\/v2\/chapters\/57\/revisions\/120"}],"part":[{"href":"https:\/\/ebooks.inflibnet.ac.in\/lisp7\/wp-json\/pressbooks\/v2\/parts\/3"}],"metadata":[{"href":"https:\/\/ebooks.inflibnet.ac.in\/lisp7\/wp-json\/pressbooks\/v2\/chapters\/57\/metadata\/"}],"wp:attachment":[{"href":"https:\/\/ebooks.inflibnet.ac.in\/lisp7\/wp-json\/wp\/v2\/media?parent=57"}],"wp:term":[{"taxonomy":"chapter-type","embeddable":true,"href":"https:\/\/ebooks.inflibnet.ac.in\/lisp7\/wp-json\/pressbooks\/v2\/chapter-type?post=57"},{"taxonomy":"contributor","embeddable":true,"href":"https:\/\/ebooks.inflibnet.ac.in\/lisp7\/wp-json\/wp\/v2\/contributor?post=57"},{"taxonomy":"license","embeddable":true,"href":"https:\/\/ebooks.inflibnet.ac.in\/lisp7\/wp-json\/wp\/v2\/license?post=57"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}