Interactive Watermarking Environments
.	GMD - German National Research Center for Information Technology Jana Dittmann und Mark Stabenau Ralf Steinmetz IEEE Multimedia Conference '98...
	Contents  1. Motivation 2. Digital Watermarking  3. Novel approaches to interactive watermarking environments  3.1 Approaches to interactive watermarking environments for single images 3.2 Approaches to interactive watermarking environments for video 4. Conclusions  References

Abstract

This article addresses one major security demand in digital marketplaces: copyright protection. Our goal is to present interactive tools which strengthen the producers? acceptance to use digital watermarking techniques to offer their data in a more secure way in the digital marketplace. Though security is recognized as an important issue in multimedia it is, ironically mostly not presented by most of the new media. Our intentions are based on the lack of media support and our focus is on the definition of new design criteria for digital watermarking in multimedia environments. The paper gives a short introduction to digital watermarking techniques, their robustness criteria, the security risks and visual interactive handling support essential for user acceptance. Furthermore, we describe four prototype tools as parts of an interactive multimedia watermarking environment for digital data: the two LabellingEditors for images and video, the 3D-Watermark and the ObjectTrackingEditor.
 

 1. Motivation

Recently security has become one of the most significant problems for spreading new information technology. The security requirements for electronic commerce and trading on a digital marketplace are essential factors to dispel acceptance problems on the part of the producers of multimedia data to offer the digital images, video or audio material in digital storages. The digital representation can easily be copied and multiplied without information loss, so that the video producers see to providing access control mechanisms to prevent misuse and theft of such material.

This article addresses one major security demand in digital marketplaces: copyright protection. Our goal is to present interactive tools which strengthen the producers? acceptance to use digital watermarking techniques to offer their data in a more secure way in the digital marketplace.

Though security is recognized as an important issue in multimedia it is, ironically, mostly not presented by most of the new media. Generally, common multimedia security mechanisms are not realized by using multimedia tools applying security. Usually the security algorithms are seen as background processes, invisible to the user. Our intentions are based on the lack of media support and our focus is on the definition of new design criteria for digital watermarking in multimedia environments.

We begin with a short introduction to digital watermarking and show what a digital watermark means. We present general existing technical approaches for digital watermarking and show application areas where digital watermarks can be used for copyright protection, customer information and embedding metadata. Beside the advantages, we show the acceptance problems and risks of existing techniques: robustness, quality loss and lack of visual interactive handling support of watermark embedding and retrieval. Visual interactions can strengthen the acceptance to use the new technology and overcome security lacks like robustness and quality loss in the watermarking algorithms when selecting appropriate watermarking features.

Furthermore, we describe four prototype tools as parts of an interactive multimedia watermarking environment for digital data: the two LabellingEditors for images and video, the 3D-Watermark and the ObjectTrackingEditor. The LabelingEditor is a graphical interface to support the insertion process of different types of label information into video streams, as well as the retrieval of such information. The 3D-Watermark gives an intuitive view to the embedded watermark. It shows the quality loss and provides information about the robustness and efficiency of the applied watermarking algorithm. The ObjectTrackingEditor is a tool to support direct watermarking of automatically or manually selected objects in multimedia data. Finally, we assess our achievements so far, and provide an overview of further work.
 

There are two major approaches of copyright protection: adding visible, perceptual marks or embedding invisible, imperceptual (hidden) information using steganographical schemes. Digital watermarks are additional hidden information, labels, holographically inlayed in the multimedia data and perceptually undetectable (invisible). Those concepts result from a direct work on the data. Ideally it is impossible to remove the embedded information (owner identification, customer information or other information about the multimedia data) without damaging the data. For example, there are two different approaches for image data: insertion of text or image components. The information which is embedded is described as hidden information or label, secret or identification code. Its principal aim is to prove undoubtedly the ownership of the data. Most approaches use secret keys for retrieving the embedded information, [14], called private watermarking. Some other approaches do not need secret keys for the retrieval, [7]. These algorithms are called public watermarking and are used for public identification of the source, author, creator, owner, distributor or authorized consumer of the data.

Summarizing, digital watermarking intends to enable the proof of ownership on copyrighted material, detect the originator of illegally made copies, monitor the usage of the copyrighted multimedia data and analyze the spread spectrum of the data over networks and severs. It should be noted, though, that the embedded signaling can be used for a variety of purposes, other than copyright control. In a distributed video production environment, for example, watermarking can be applied to integrate different kinds of data into the video material, such as customer information, or meta data as a format independent part of the video, e.g. information about cuts or scene descriptions.

Requirements - robustness criteria: Currently we have concentrated our work on digital image and video data. There are a number of characteristics and requirements for watermarking mechanisms:

• difficult to notice, prevention of visible loss of quality of the multimedia data.
• no loss of relevant information
• private copyright information must not be detected by anyone but the copyright owner himself and an independent control instance by using a secret key (private labeling)
• public copyright information must be accessible for everyone to call attention to the copy- and reproduction rights.
• compatibility with the original data
• Changeable and multiple labeling, allowing modifications of the label and multiply labels to co-exist for tracking the content from manufacturing to distribution to eventual sales, where each point of the production and distribution chain can embed their own unique label information.
• Robustness and resistance against image processing, geometrical distortion, signal transformation, digital-to-analog or analog-to-digital conversion, and lossy compression.
• In order to prevent any copyright forgery, misuse or violation the technique is to provide security and robustness of the embedded label against a variety of threats which include:
• Detecting embedded locations by comparing different labeled versions of the same original material
• Finding and altering the embedded label through visual or statistical analysis
• damaging or removing the embedded label using common multimedia processing, such as lossy JPEG compression, color space conversion, and low pass filtering.
Security problems: Today the existing labeling techniques have different security problems. We have tested several watermarking techniques to proof the robustness of the algorithms, [9]. The following security problems were detected:
• Transformations like rotations, stretching, shrinking, shifting or scaling of the image data generally caused a damage of the label. Scaling and cutting operations were more difficult to handle than the other geometric transformations because of the loss of fine details. Current algorithms do not ensure that the relevant objects are watermarked robustly, they spread the watermark of the whole image and gathering quality loss. There is no guaranty that the relevant objects contain a sufficient amount of watermarking information for a later retrieval.
• The pixel-based approaches did correct retrieval badly when modification of the content like lossy data compression, for example JPEG compressions, were performed. These techniques could not retrieve the correct label if the loss was more than 40% where the original image was unknown. The DCT based algorithms are more stable and could retrieve the correct label up to 50% loss.
• Physical damaging of the image material like cutting out pixel lines and edges of the image are general problems of the algorithms, [9].
Based on this knowledge watermarking editing tools should consider the security problems to proof the robustness of the used techniques. The possibility to evaluate the properties of the embedded watermark and to use alternative algorithms can convince the copyright holder to use the new techniques anyhow.

Transparent watermarking: Additionally, our latest experiments have shown that the user interaction for labeling image data is difficult, non-uniform and inconceivable. The user has no possibility to understand and interact with the watermarking process. He cannot proof the robustness or see what happens with his work. He cannot measure or influence the quality loss. Important parts of the video can not be selected separately to strengthen the robustness of the watermark. Cutting out not important parts would not cause a fault watermarking retrieval. Only the important part should be watermarked. Because watermarking works directly on the media material the authors and producers should be involved in the labeling process to observe what happens with their data and to dispel acceptance problems of the labeling process, too. Thus, one major part of our work is the design of visual interaction tools for labeling and the design of metaphor to visualize labeled multimedia data.

Our goals is to design interactive watermarking environments as an enabling technology for the use of the watermarking technologies in digital environments and to make the security algorithms more attractive and usable.
 

Our research seeks to address the insights gathered from the above discussion to provide new design criteria for digital watermarking mechanisms considering visualization techniques and multimedia system technologies. Based on research on GUIs, [2], [3], [6], [11], [20], we focus on the development of visual authoring systems embedding label information, such as:

• Visualization of the watermarking procedure and of the caused quality loss,
• Testing robustness of the incorporated watermark,
• Supporting interactive labeling, where most characteristic objects of the multimedia data are labeled with a first automatic suggestion of characteristic objects using object tracking mechanisms,
• Solutions for the labeling and retrieval process separated for private and public labels,
• Categorization of embedded labels in pure copyright information, customer information and additional metadata,
• Monitoring and spread spectrum analyses of the copyright data of networks,
The goal is to develop interactive tools for intelligent access and navigation on the watermarking information. In this section, we present two tools for single images and two tools which support the above goals in a distributed video production environment, currently under development at GMD. Please note that the current prototypes only address image and video data. Further research is needed to provide solutions for audio and 3D-objects.
 

Our first goal is to support the labeling process itself and the annotation with copyright information. Therefore we have developed a LabelingEditor for single images, demonstrating the labeling process and the robustness of the watermark. Second, we want to measure the quality loss and the robustness in a more intuitive way using 3D-elements, forming a 3D-Watermark as a visualization for image watermarks.
 
• LabelingEditor for single images

The LabelingEditor for images provides support for labeling and retrieval processes as well as for robustness tests. The editor is completely written in Java. The copyright holder has two views on the data, the original and the watermarked one. Thereby he can visually compare the images and get an impression of the visual quality loss. To get a more detailed view on what happens with the original ones, the changes made during the watermarking process are visualized, showing the difference image.

Labeling and retrieval: Currently, two different kinds of watermarking are used and can be configured in the editor: a amplitude-modulation-based and a DCT-based algorithm, [13], [10]. The "File" menu offers the open function for original images or already watermarked images. The original image is presented on the left, the watermarked image is presented on the right, see figure 1.

Figure 1: LabelingEditor for single images

Embedding and retrieval are controlled by the two control buttons. The scrolling area shows the detailed embedding or retrieval process steps. If the original image is available the copyright holder can switch between the two views: showing the watermarked image or the difference view using the radio buttons. The difference view provides a more illustrative description of what happened with the original one. Black areas stand for no changes made during watermarking process, the other colors show the changes that were made. A more detailed view can be seen in the attached demo video.

Robustness test: Coupled with the embedding and retrieval, the editor provides basic robustness tests: geometrical transformations like rotation, scaling, cutting or lossy compression and low pass filtering. The editor performs the distortions and simultaneously checks, if the watermark information can be retrieved correctly. The behavior is demonstrated in the demo video.
 

• 3D-watermark

Another novel approach uses 3D-objects to visualize the watermarked image. Based on the experiences made with the LabelEditor for single images, we have approved the view to the watermarked image. The idea is to transform the difference into a 3D scene, (if the original image of the copyright holder is available). The 3D-Watermark is a Java-based application which generates a VRML scene viewing the changes made in the original image. A 3D relief appears, figure 2:
 
Figure 2: 3D-watermark

The following can be seen:

• the absolute difference between the original image and the watermarked image,
• areas that are watermarked,
• the intensity of the watermark, measured by the height of the 3D relief ,
Based on these information:
• the user gets an intuitive understanding what a watermark means,
• the quality loss can be measured,
• it can be seen, if relevant image objects are watermarked,
• the robustness can be evaluated, regarding the intensity,
• different algorithms can be compared.
This knowledge can be used to change the watermarking parameter or to choose another watermarking algorithm. Furthermore, the watermarking algorithm can be tested and improved. The need for ObjectTracking is one result of this evaluations, see chapter ObjectTracking editor. Compared to the background the example shows only few distortions in the face region.

To improve the information about the visual distortions the 3D-Watermark can visualize not only the absolute differences. As an alternative the 3-Watermark presents the differences separated into the RGB or HSV model, figure 3:
 
a) red-channel
 
b) green-channel
 
c) blue-channel
 
d) hue-3D-watermark
 
e) saturation-3D-watermark
 
f) value-3D-watermark

Figure 3 a)-f): RGB and HVS models

The RGB model is hardware-oriented. The red channel was less used than the green and blue. By contrasts, Smith?s HSV (hue, saturation, value) model, [17], is user-oriented, being based on the intuitive appeal of the artist?s tint, shade, and tone. The RGB watermarks show a more smooth relief than the HVS watermarks. Especially the hue and saturation evaluation show extreme alterations. Our next work is to combine the watermarking embedding and retrieval steps directly with the 3D-Watermark output to get a visual feedback, what happens during these processes, and to see which values are influenced or needed for retrieval.

Because of the statistical properties of a watermark, the 3D-Watermark cannot visualize all embedded data. It could be possible, that information is embedded although no visible distortions are observed, because the image data properties were already in the correct meaning. This disadvantage does not influence the review of the quality loss and the visual distortions, but can affect the review of the robustness based on the intensity of the 3D relief. To check this, the LabelEditor can be used to cut out the appropriate sections and try to retrieve the watermark again. Compared with the image background the example shows few distortions in the face region and so it seems to be not robust against cutting the face region, figure 4:
 
Figure 4: 3D-face-region

The results in the LabelEditor show that there is really a security lack, figure 5:

Figure 5: retrieval in the face region

Our experiments showed that this drawback does not appear very often. But, to improve the 3D-Watermark we will offer an additional view to the actually used pixel of the image for retrieval. For security reasons, this option is only accessible for copyright holders. Of course, this additional view cannot provide information about the visual quality loss compared to the original.
 

Our first goal is to support the labeling process in distributed video production environments. The video data is located on a digital video server to support extensive annotation and retrieval with copyright information, customer information and metadata. Second, we want to support selective watermarking of relevant video objects to strengthen the robustness. Currently we have addressed MPEG-Video.
 
• VideoLabelingEditor

In distributed video production environments video data are located on a digital video server. The video placed on the server needs to be annotated with copyright information, customer information, and special metadata which support the retrieval of the video sequence or parts of it. Once the material has been annotated, a customer may use the additional information for pre-selection queries, for example:
• to ensure the originality and copyrights of the required material, or
• to search for particular meta information, before the customer makes a decision to buy and download the sequence.
Our goal is to support the annotation process executed by the copyright owner, as well as support the retrieval and search processes done by the user, by providing a graphical user interface: the LabelingEditor, also called MetaDataEditor. A special log in procedure allows the copyright holder to structure the required label information in private and public information. If the log-in procedure is used with a secret key, the information is protected, which means that other users need the key for accessing the additional information. In all other cases, the information is publicly accessible. The LabelingEditor consists of two parts: the editor itself, the MetaDataEditor, and a visualization component for the video sequences, as suggested by the two windows in figure 6.
 
Figure 6 VideoCube and LabelingEditor

We now discuss the components of the LabelingEditor and their functionalities. We begin with the Editor, and then describe the Visualization Component.

The editing tool: The editing tool actually embodies two distinct features. For the copyright holder (the annotator) it provides the facilitation for labeling. For security reasons this feature is only provided by log-in with the secret owner key. All other users, including the annotator, can use the editing tool only as a retrieval interface.

The tool offers different views to copyright, customer or meta data for each frame of the video. The labeling process can be configured dynamically by selecting the watermarking mechanism, a DCT-based and a amplitude-modulation-based algorithm. It must be stressed, however, that the current stage should merely be understood as a first step towards a more complete watermarking scheme, which supports all kinds of existing watermarking techniques. At present, we are also working on a new watermarking mechanism in which we seek to consider the advantages of the existing techniques by overcoming a number of shortcomings, e.g. increasing the amount of label data that can be embedded into image data.

In order to insert all information of the Editor, we use a special syntax description and data reduction technique. Before the label data will be embedded, the information is separated and transformed into the watermarking syntax, as described in figure 7.
 
Figure 7 labeling process

To differentiate into public and private data, the private data P will be encrypted with the secret owner key k and the encryption F to retrieve the final watermark W: W = F_k(P). The public data remains unencrypted and accessible. To reduce the data volume the label data is Huffman-encoded. We use watermarking techniques where the watermark is inserted into perceptually significant regions of the image, [10]. Depending on the image characteristic I the watermarking information W is embedded and the labeled image I_W is created:

I_W =I + f(W,I) (public watermarking)

I_W =I + f(F_k(P),I) (private watermarking)

Depending on the watermarking technique the function f modifies the image data for embedding the label.

The label can be integrated into a single frame of a video, or can be spread over all frames. For the latter, though, a removal of frames might lead to a partial or complete loss of the label data. However, the advantage of this approach is that more data can be embedded.

For a transparent labeling process, the copyright holder gets the pixel positions where the label is embedded into the video frames.

The visualization component:

The video visualization component provides the view inside the video sequence (see the left window in figure 6). We use a video visualization in the form of a 3D-cube, [18]. The frames of a selected video are displayed as a floating 3D block, where the current image is represented as full image, while older images form the lateral surface of the cube. Thus, the 3D-Cube represents temporal and content aspects of the video sequence. Features, such as editing rhythm, shot boundaries, camera motion, etc., are clearly visible in the pattern appearing on the surface of the cube. The user can stroke across the cube with the cursor to access any part of the video, even a single frame, like thumbing through pages of a flip book.

The Visualization Component is synchronized with the Editor. This means that while the user navigates through the VideoCube, the editor shows the annotated information. The Visualization Component is activated after a user has specified a query in the Editor. The retrieved video frames are highlighted with a distinct color to indicate their position in the video sequence.

The visualization component can be exchanged to offer alternative presentations, e.g., to the traditional video player or a linear two dimensional presentation.
 

• ObjectTrackingEditor

Current watermarking techniques do not allow the copyright holder to select important parts of the video or of single video frames. The user has to trust the watermarking algorithm in two aspects: firstly, which objects or areas of the video are labeled, and secondly, where the label is located. If it is possible to select relevant objects separately, the watermarking algorithms are applied to this objects separately and strengthen the robustness, for example, against cutting, when the user cuts out relevant objects of the video the watermark is also cut out. Remember, current algorithms do not ensure that the relevant objects are watermarked, see chapter LabellingEditor for single images and the 3D-Watermark.

Our goal is to support transparent and selective labeling. This means, characteristic objects of the multimedia data are pre-selected with an automatic object tracking algorithm based on the MPEG-4 standard, [16]. Such object-tracking relies on ?object-based? descriptions of video sequences in terms of semantic relations between objects or regions. The description process of video objects may be performed automatically, semi-automatically or manually, i.e. as part of the video production process. Automatic processes search is described in [19]. Semi-automatically techniques use interactive pre-selected view points to recognize an object and use these points to perform object tracking, [15]. Manual techniques require user interaction to find objects and perform object tracking.

Once video objects have been located in a video sequence, a tracking procedure can start to calculate the positions of the video objects in successive frames and then find the trajectory of the objects throughout the whole video. Content-based interactivity is thus based on the data structure, and independent of the coding techniques used for each accessible unit. In such a way, the user is provided with suggestions about video objects that might need watermarking, based on a pre-parsing of the video. The actual labeling process is not started before the user has agreed on the selection, providing him or her with the possibility to deselect parts, or select additional other part of the video according to his interest. A possible outcome of the object identification process is suggested in figure 8, where the left frame shows the original image and the right frame the identified objects.
 
Figure 8 identification of objects

For our automatically object tracking in video sequences we use fuzzy contour refinement, developed by Steudel (1997), [19]. This algorithm provides good results for different kind of test sequences compared to ordinary motion-based approaches. For the semi-automatically object detection we are using algorithms for edge detection based on the Canny techniques, [4]. Currently we combine the object-tracking algorithm with our watermarking techniques described in Dittmann et al. (1998), [10]. We apply these techniques in the following way:

• The objects are pre-selected and predicted automatically
• The user can confirm the selection or can deselect parts.
• The selected parts are watermarked within the smallest surrounding rectangle.
Figure 9 illustrates the selection process for semi-automatically object tracking in the watermarking editor.
 
a) Original video and selected objects
 
b) Detected objects with Edge-Detection

Figure 9 Watermarking editor with object identification

Our new watermarking mechanism support direct watermarking of automatically, semi- automatically or manually selected objects. We combine the advantages of Caronni?s approach, [5]combined with the Fridrich approach, [12], using rectangle tagging by modulating the brightness of the objects for later retieval and the approach of Koch et al. (1995), [14], using DCT coefficient manipulation to embed robust labels. The retrieval process starts with a search inside the data and tries to find watermarked objects searching for a typical watermarking pattern. All objects will then be checked with an equivalent retrieval process and the embedded information is retrieved.
 

In this paper we have discussed watermarking techniques, their possibilities and disadvantages to ensure copyright protection. We pointed out that current watermarking techniques lack the facilitation of sophisticated multimedia techniques for the embedding, authoring and retrieval of label data. Our goal was to design interactive watermarking environments as an enabling technology for the use of the watermarking technologies in digital environments and make the security algorithms more attractive and usable. Thus, we illustrated novel tools for embedding data, i.e. the LabelingEditor for image and video data, the 3D-Watermark, and the ObjectTrackingEditor. Each tool and its theoretical foundation is best regarded as a platform that demonstrates the possibility of interactive multimedia authoring/retrieval-oriented watermarking.

At present, we are engaged in research into the design of a framework for distributed, digital video production. The aim of the framework is to provide support for all stages of the video production process.

Furthermore watermarking not only enables to prove the ownership on copyrighted work and to detect the originator of illegally made copies, but also to analyze the spread spectrum of the data over networks and severs. Our recent work is focused on a MonitorAgent that resembles, in parts, a search engine that provides the copyright holder with results about where, by whom and for what purposes the work is used. The visualization of monitoring results might be achieved in different ways. The MonitorAgent performs an analysis on the usage of the detected material using a 3D-metaphor.