OntoLog |
Ontology-Based Annotation of Video and Audio |
Start OntoLog! (Oracle version) |
||||
|---|---|---|---|---|---|---|
|
||||||
User Manual |
|
OntoLog is a Java application, and requires Java 2 v1.4 (or later) and QuickTime for Java or Java Media Framework (JMF) to run. I recommend QuickTime for Java, if your media is QuickTime-compatible, as it is more stable and compatible, as well as less error-prone, than JMF.
To install the latest version of Java, just follow the instructions on this page. For best results, install the latest version before you install QuickTime for Java or Java Media Framework. You can use OntoLog without installing any media player, but you won't be able to play media or create any temporal annotations. For the example project, JMF 2.1.1a works best.
To install QuickTime for Java, download and install QuickTime as normal, but when the installer asks if you want a Minimum, Recommended or Custom installation, choose Custom. In the next panel, make sure "QuickTime for Java" is selected, as well as "QuickTime Internet Extras", if you want OntoLog to be able to use mp3s.
If you had QuickTime already installed, use the QuickTime Updater. Use the "Custom" button instead of the "Update Now" button, and make sure "QuickTime for Java" is selected, as well as "QuickTime Internet Extras", if you want OntoLog to be able to use mp3s.
To install Java Media Framework, download it from this page. The latest versions of JMF have removed mp3 support, so if you want to use mp3s (like the example project does), you have to either use QuickTime instead, or use version 2.1.1a (Windows version) of JMF. For the example project, JMF is best, as QuickTime seems to have some difficulty playing mp3s over an HTTP connection. In general, though, Quicktime is more stable, and supports more media formats.
To start OntoLog for the first time, click on the "Start OntoLog!" link in the top right corner of these pages. I everything goes according to plan, this will activate Java Web Start, which will start to download the OntoLog code. This may take some time, depending on the speed of your Internet connection; it is about 5.4 MB. Then Java Web Start will inform you that OntoLog requires full access to your computer, and warn you that it doesn't quite trust the code (this is only because I haven't bothered to pay a trust company to vouch for my digital signature). I'm 99 % sure that OntoLog won't damage your computer, but I can't be responsible for any freak accidents that might happen if you use it. If you trust it, OntoLog will eventually present you with a login dialog.
I have made a sample database called "ontologdemo", to which you can log in using the user name guest and no password. The database contains a sample project called "Jon's Gloria demo project", containing a few mp3s annotated using a some custom ontologies from music domains, and metadata conforming to the Dublin Core standard. You'll need a reasonably fast Internet connection to use this database; ISDN may be too slow for comfort. Even ADSL makes you wait a little, so if you want to seriously try OntoLog, I recommend that you use a database on your local network or computer. I can help you set up your own MySQL database server, if you want.
The guest user has read-only access to the database any changes you make won't be stored in
the database. If you want write access, you can go about it in two ways:
There is also another example project you can use, containing recordings of lectures in the course "Information Technology, Introduction" from the autumn 2003 at NTNU. The database name is itgk, or itgk_en if you want the annotations in English instead of Norwegian. The user name and password is the same as above.
You can always start OntoLog again by clicking the "Start OntoLog" link. It won't download the code again (Java Web Start caches it), and it won't ask you if you trust the code again (Java Web Start remembers that you do). There are also two other, equivalent methods: You can use the Java Web Start program, which should be somewhere on your start menu; and you can have Java Web Start create shortcuts to OntoLog on your desktop and/or start menu. As default, Java Web Start will ask you if you wish to have such shortcuts created when you start OntoLog. You can also have them created later through the Java Web Start program.
OntoLog will be automatically updated by Java Web Start when new versions are available (provided you have an Internet connection). Updates will in general be much faster to download, since the bulk of the 5.4 MB you have to download the first time, is database drivers and libraries that don't change very often. The main OntoLog program is just about 340 KB.
As a goodly-sized Java application, OntoLog needs a reasonable amount of memory and processor power. I use an 800 MHz Pentium III with 512 MB of memory; anything less than 400 MHz and 128 MB may be too weak. To use databases and large media files on the Internet, you also need a fast network connection a LAN is best. However, OntoLog doesn't require network connection: You can install your own database locally, and work on local media files. The MySQL database is free, and can be downloded from www.mysql.com.
The current version of OntoLog is 1.6. Version 1.4 is not publically available, because it was a bit too buggy. The data format was changed with versions 1.3, 1.5 and 1.6, so databases created with these versions don't work (well) with earlier versions of OntoLog. However, databases created with earlier OntoLog versions are converted when you open them with a newer version of OntoLog. If you have any trouble when a new version comes out, let me know; I can probably fix it.
The quickest way to learn to use OntoLog is probably to experiment with it. I have provided an example database (see above) for that purpose. The database is read-only, so if you try to change anything, you will often get an error message. In some cases, you may be able to do some apparent changes, but these will not be stored in the database, and the next time you log in, the changes will be gone.
The database contains a single project, "Jon's Gloria demo project", which you can open with "Open Project..." on the File menu. If you the select "Project Properties..." from the same menu, a window appears with information about the project. Note that the only purpose of this project is to be an example I haven't given thought to whether this particular kind of media annotation is useful or not.
In the Project Properties window, a table near the top shows the project's properties. You may add and remove properties, and edit them by typing in the Value column. Such tables are found several other places in OntoLog as well. Below the table is a list showing what schemas the project uses. Schemas decide wich properties may appear in the property tables; more on this below. The bottom list shows the ontologies is used by the project. The ontologies are used to annotate media by associating intervals in the media stream to concepts in the ontologies.
In the main window, in the Media panel, you can see the four "media resources" (in this case mp3 files) the project contains. The main purpose of OntoLog is to annotate (describe, index) these. They are all described by several properties, shown in the table on the right. To play a media resource, select it and press the Play/Pause button on the media strip at the bottom of the window. Alternatively, double-click on the media resource. It may take some time for playback to start, as the mp3 file must be transferred to your computer. "Quoniam Tu Solus Sanctus" is the smallest file, but "Propter Magnam Gloriam" is almost as small (838 KB), and a bit more interesting, in my opinion.
In the Ontology panel, you can view and edit the ontologies and their properties. Ontologies are hierarchical collections of concepts (terms) relevant to some domain. The example ontologies are a small taxonomy of chorus parts, an arrangement of some of the participants of the concert the music was recorded on, an ontology about the keys and tempos in musik, and the Dublin Core Element Set. These are used for two purposes: to annotate the media resources by connecting concepts to media intervals e.g. the Soprano concept is connected to all the intervals where the sopranos in the choir are singing and to describe the media resources and ontology concepts with properties and values from the ontologies.
The Logger panel is where these temporal annotation intervals are created, viewed and manipulated. On the left are the ontology trees; on the right is a timeline display, showing the annotation intervals as horizontal black lines, and the playback time as a pair of vertical, red lines. If no lines are shown, select one of the media resources from the Media panel, or the "Media Resource" drop-down-box at the bottom of the window. The line corresponding to the "Chorus parts" ontology will probably be irregularly thick, and cover most of the timeline display. This is because it is actually an aggregation of the interval lines associated with its member concepts. The thickness corresponds to the number of concepts in the ontology that are "active" at any given moment. Try expanding different parts of the ontology trees and note how the black lines behave. Hopefully, it will soon make sense.
| These two together... | ![]() |
| ...becomes this. | ![]() |
The Logger panel can also show some simple statistics on the intervals: The number of intervals per concept, their total length, and the percentage of this length with regard to the total length of the media resource. This information is also aggregated according to how you expand the ontologies on the left. Select the "Show Statistics" checkbox to display the statistics table.
The Resource Description Framework (RDF) is a standard for describing resources ("resources" can in principle be anything at all). OntoLog uses RDF for almost everything, so it is practical to know one or two things about RDF.
In RDF, information is represented as statements consisting of a subject, a predicate and an object. The subject is the thing (resource) that is being described, the predicate is some property of the subject, and the object is the value of the property for the given subject. For instance, if I were to express that this web page is written by me, the web page would be the subject; "written by", "author", "creator" or something like that would be the predicate; and I (some representation of me, e.g. my name) would be the object. A few examples, in tabular form:
| Subject | Predicate | Object |
|---|---|---|
| http://www.idi.ntnu.no/~heggland/ontolog/manual.php | author | Jon Heggland |
| http://www.idi.ntnu.no/~heggland/ontolog/manual.php | lastUpdated | 2002-08-27 |
A resource is something that can be identified by an Universal Resource Identifier (URI). All URLs (web addresses) are URIs, and OntoLog is geared towards using URLs, so feel free to substitute "URL" for "URI" in the following discussion (and in OntoLog in general).
Only resources can be RDF subjects, so everything you describe in OntoLog (and RDF) must have an URI. The resources need not be electronic documents or entities, as long as you can find (or make up) an URI to identify it. For example, I could represent myself with the URL of my homepage (though that would be ambiguous if I also want to describe my homepage), an URI based on my employee ID number or something like that. OntoLog will usually generate URIs for you automatically, based on the titles of the projects, ontologies and concepts you create, so you don't really have to worry about it if you don't know what an URI is.
Objects may be either resources or literals. A literal is quite simply a piece of text. Predicates are resources (a special kind of resource named Property), which means that they, too, can be described like any other resource. There are two special properties, domain and range, that you can use to describe other properties in order to restrict what kinds of resources may be used as subject and object in statements using the property as predicate; more on this in the schema section.
For more information on the RDF model, see the World Wide Web Consortium's RDF Primer.
Projects are used primarily for two things:
There really isn't much more to say about projects that hasn't already been said under Understanding the Example.
The properties used to describe all the resources in OntoLog are defined in ontologies. Each property may optionally specify what kind of resource it may be applied to; if it doesn't say anything, any kind of resource may use it. They may also specify what kind of objects (values) they may take.
OntoLog provides two ontologies as default: the Dublin Core Element Set v1.1 (DCES), and the Dublin Core Element Set Qualifier Library. DCES is a widely used metadata standard, and contains fifteen properties: title, creator, subject, description, publisher, contributor, date, type, format, identifier, source, language, relation, coverage and rights. See http://www.dublincore.org/documents/dces/ for descriptions of these properties. Unlike most ontologies, DCES only defines properties, not any classes or members. The Qualifier Library contains specialisations of some the DCES properties; for instance "Is Part Of" and "References", which refine the DCES "relation" property. It also defines some data type classes, but these are not of much use in OntoLog. Dublin Core is used as default in all new projects; if you want to use the Qualifier Library as well, you have to add it from the Project Properties dialog. All the Dublin Core properties are applicable to all kinds of resources.
When you try to add a property to a resource by pressing the "Add..." button next to a property table, a dialog like the one on the right appears. This contains a tree list of all the properties applicable to selected resource, grouped by ontology. The example project uses four ontologies, but only two of them define any properties applicable to a media resource, which is what we're trying to add properties to here. The DCES properties can be used on any kind of resource, so they will always show up in this dialog, while "Has key", "Has tempo" and "Lyrics" can only be used on media resources. Therefore, when you add properties to projects, ontologies, concepts and intervals, those properties won't appear in the dialog.
Next to most of the property tables, between the "Add..." and "Remove" buttons, there is a button calles "Add Usual". When you press it, OntoLog will add the properties you "usually" use for this kind of resource, to the resource shown in the table. The way this actually works, is that OntoLog keeps track of what properties you use to describe for instance media resources. If a property (e.g. Date) is used for more than half of the media resources in the project, OntoLog regards it as "usual" for media resources, and adds it to the selected media resource when you press the button. If the button is inactive, it means that either you don't have any "usual" properties, or that the selected resource already has all the usual properties.
Media resources are the video or audio files that OntoLog helps you annotate, index and analyse. OntoLog can use both QuickTime and Java Media Framework (JMF) for media playback, so quite a few formats are supported. This list shows QuickTime's capabilities, and this one shows JMF's.
To add a media resource, click the leftmost button on the Media toolbar (after opening or creating a project). This will prompt you for an URI, which must point to the media file. You can browse the files on your local computer using the "Browse..." button to the right of the URI field, creating a file-URL. HTTP URLs are also possible the example projects uses HTTP. Other kinds of URLs may also work, depending on JMF. You can also drag a media file from the operating system (from Windows Explorer, for instance) and drop it over the media list.
You can remove media resources from the project by selecting them and pressing the trashcan button. This will not delete the actual media file, but will remove it from the project. All its properties and annotation intervals will be deleted.
The same media resource may be part of several projects, but its properties and annotation intervals are local to each project. Removing a media resource from one project has no impact whatsoever on other projects.
Ontologies are at the heart of OntoLog's annotation system. The basic idea is that you can record the presence of some concept in a particular piece of the video or audio stream. This concept can represent anything at all a real-world entity like a person, an object, or a location; or an event, a camera technique, a plot device; or more abstract things like a news category or a political viewpoint. Exactly what you choose to represent with concepts will depend on the media you are annotating, and your purpose in annotating them. Ontologies are collections of concepts, organised in a non-rigid hierarchy.
Designing and using ontologies is a science in itself. Here is a good introduction to ontologies and how to design them (pdf, 501 kB).
There are two kinds of concepts in OntoLog's ontologies: classes and members. Classes are shown as blue discs with a "C" inside, while members are red discs with the letter "m". Conversely, there are two kinds of hierarchical relations, subclass and member. The difference between them is subtle, but significant in some ways.
Conceptually, the class-subclass relationship is a set-subset relationship. The set of "things" represented by a subclass is a subset of the set of things represented by its parent(s). An example from one of the example ontologies: The set of sopranos is a subset of the set of female singers all sopranos are female (as this ontology doesn't consider boy sopranos). The class-member relationship, on the other hand, represents the set-member relationship. One particular soprano (say, Kirsten Flagstad) would be a member of the set of Sopranos, not a subset.
From a more technical point of view, (sub)classes and members are different in that members may not have "children" they only appear at the bottom of the hierarchy. Also, members are never top-level concepts; they must have a class to be member of.
There is another, more advanced feature concerning classes and members. Like in object-oriented programming, classes are considered types, and their members are considered instances of this class or type1. This may be important if you use custom properties. You could, for instance, create a property (say, phone number) applicable to the class Person, a concept in your ontology. OntoLog would then enable you to describe members of Person with the phone number property.
For more abstract concepts, it may be hard to apply the distinction between classes and members. If that is the case, you may want to just use classes and subclasses, as that is the least restrictive relation.
To create an ontology, select "Create Ontology..." on the Ontology menu and enter a title. OntoLog will generate an URI for you; if you prefer to specify the URI yourself, deselect the "Generate identifier (URI) automatically" checkbox in the dialog. Ontologies usually have URIs that end in the character '#'. To add concepts, select an ontology or concept and press one of the two "Add" buttons on the Ontology toolbar. Again, OntoLog will prompt you for a title and generate the URI automatically. Concept URIs are usually made by appending a local identifier to the URI of its ontology; see the example ontologies for examples of this naming scheme.
You can reorganise ontologies by dragging concepts. You can drag them to change the order of subconcepts, or to change a concept's parent. You can't drag them just anywhere, however; you cannot make a concept a subconcept of itself or its descendants, drag a member to the top level or move a concept to another ontology.
A concept can be a child (subconcept or member) of several different concepts at once. The example class Soloist, for instance, is a subclass of both Person and Performer. In other words, it is a subset of the intersection of the Person and Performer sets. To make concept a child of several concepts, you can add it several times (make sure that you enter the same URI every time), or you can drag it to another parent while pressing the Ctrl key. The mouse pointer will change to indicate that you are performing a kind of copy operation. Again, OntoLog will prevent you from performing copying operations that it judges unworthy.
To delete a concept, select it and press the trashcan button ("Remove ... from ...."). If the selected concept has several parents, it will not be deleted, just removed as a child of this particular parent concept. For instance, the Soloist concept appears twice in the Participants ontology tree, under Person and Performer. If you select Soloist under Person and remove it, Soloist will no longer be a subconcept of Person; however, it will still exist as a subconcept of Performer. If you then select Soloist under Performer and remove it (and confirm the deletion dialog that appears), the Soloist concept and all the data related to it in this project will be deleted from the database (since that was the last occurrance of Soloist).
You can add and remove ontologies from the project in the "Project Properties" dialog. You can also add ontologies by using the "Add Ontology..." item on the Ontology menu, and remove an ontology by selecting it and pressing the trashcan button in the Ontology or Properties panels. You can also import other ontologies by pressing the "Import..." button in the "Add Ontologies" dialog, or selecting "Import Ontology..." in the Ontology menu. This prompts you to enter the URI of the ontology, which should point to an electronic resource, specifically an RDF Schema (RDFS) written in XML. You may also import the schema from some file, using the second text field in the dialog, but you always have to enter the schema's official URI as well.
Explaining RDF Schema is beyond the scope of this manual. See http://purl.org/dc/elements/1.1/, http://purl.org/dc/terms/ and http://www.idi.ntnu.no/~heggland/ontolog/demo/music.xml for the RDFS/XML definitions of the schemas used in the example project.
Removing a schema removes it from the project, but not from the database. You may add it again later, and/or use it in other projects. The schemas also determine which properties are shown in the property tables. If you remove a schema, the properties defined in it will disappear from the tables, but they are not deleted, just hidden. If you re-add the schema, the properties will again appear.
Different projects may of course share ontologies.
OntoLog provides a separate panel for creating and editing properties. To create properties, select a schema (or a property in an ontology) and press the two "Add Property" button on the toolbar. OntoLog will prompt you for a title and generate the URI automatically. Property URIs are usually made by appending a local identifier to the URI of its ontology, separated by a '#' character, though the Dublin Core ontologies don't follow this naming scheme.
Properties may be described with properties, just like any other resource in OntoLog. The Properties panel includes normal description panel for this purpose, with controls for two particular properties: Domain and Range.
The Domain property has a special function: It is the Domain that decides if the property should show up in the "Add Properties" dialog described previously. All the Dublin Core properties have the Domain <anything>, which means that they always show up they can be used to describe anything. In contrast, the "Has key" property from Jon's music ontology can only be used to describe members of the class Media Resource the media resources on the Media Panel. Possible values for the Domain property includes the OntoLog classes Media Resource, Project and Interval, as well as the classes in whatever ontologies your project is using. For instance, the Leader property in the Gloria participants ontology has Ensemble from the same ontology as its domain it is used to indicate who is the leader of the Ensembles in the ontology.
The Range property has a similar function. It decides what kind of values you can use for the property. The default value is <anything>, which means just that you can type a string value for the property, or use any object from your ontologies as property value. Possible values for the Range property are the the classes in your ontologies. The "Has Tempo" property has the Tempo class as its range,; the Leader property has Person as its range. Thus, the only possible values for any Leader description are the members of the Person class (and its subclass(es)).
Please note that changes in the domain and range are not retroactive. Any changes do not cause existing descriptions that violate the new domain and range restrictions to change or be deleted. You have to take care of that yourself (for now).
The creation and manipulation of temporal annotation intervals connected to concepts in ontologies is what gives OntoLog its name. Annotating (describing) possibly overlapping intervals in media in near real-time is often called logging, and OntoLog tries to empower this technique by employing structured ontologies instead of flat lists of keywords.
The resulting set of annotation intervals may have many uses:
Controlling media playback is an important part of the logging process. At the bottom of the window, there is a set of controls for this: A pause/play button (keyboard shortcut: Alt-Space), a button for jumping to the beginning of the media file, and a time slider.
To jump to a different place in the media file, you can use the time slider at the bottom right. In the Logger panel, you have two additional options: You can either click on the desired time in the time ruler above the interval display, or you can drag the vertical double red line representing the playback time. As a side effect, this will pause the playback. You can also skip forwards and backwards using the right and left arrow keys together with Alt, Ctrl or Shift; Alt-arrow will jump one second, Ctrl-arrow five and Shift-arrow ten seconds.
To create new annotation intervals, you must turn on logging mode by pressing the togglebutton to the left on the toolbar. Then, change the playback time to the position where you want the interval to start (or end), and click on the desired concept in the tree list on the left. Jump (or play) to where you want the interval to end (or start), and click on the concept a second time. (This is far easier to do than to explain.) During logging, intervals "under construction" will be displayed as red lines, one end of which is attached to the vertical red lines indicating the playback position. You can cancel an erroneous logging by right-clicking on the concept, and selecting "Cancel this logging" in the menu that shows up. Also, intervals with no length (created by clicking a concept on and off while the media is paused, for instance) are ignored.
The logging process is designed to be usable in near real-time. That is, you can start playback and logging at the beginning of the media file, and click different concepts on and off as they becomes relevant in the media, without interrupting playback. To make the concept selection more efficient, you can assign keyboard shortcuts to the concepts. To do this, you must turn off logging mode. Then, select a concept, press the button to the right of the "Logging on/off", and enter a single letter. Pressing this key during logging will then be equivalent to clicking on the concept. If the concept's title includes the letter, it will be underlined in the tree list to help you remember.
Sometimes, you may want to have intervals that are related to more than one concept. For instance, if you are analysing the behaviour of several interacting actors, is is natural to represent both actors and activities as concepts in different parts of an ontology, and to annotate the performance of an activity by relating its interval to both the activity itself, and to its actor(s). You can do this during logging by first logging one of the concepts (say, the activity) as normally; then, you Alt-click on the additional concept(s) you want to relate to the same interval. (You can also hold down the Alt key while pressing the concept's shortcut key.) OntoLog will then consider the interval as relating to more than one concept, and visualise and treat it accordingly. When the activity (or whatever it is you are logging) ends, you "click off" any of the concepts, and the interval will end for all the other co-related concepts as well. In the logger panel, intervals related to more than one concept will have a small icon displayed at its right endpoint, to visually indicate its special properties. Note that such intervals are allowed to overlap with other intervals related to the same concept, unlike normal intervals, who are joined with any overlapping neighbours.
OntoLog provides a similar functionality for copying interval startpoints during logging. For instance, at some point in your video, several concepts may start at the same time (though they are not guaranteed to end at the same time). Instead of pausing the video, turning on all the concepts, and then resuming the video, you can start one of the concepts without pausing. Then, you can Ctrl-click on the other concepts (or hold down the Ctrl key while using their keyboard shortcuts), and new intervals will be started with the same start time as the previously started interval. Thus, you can create a lot off annotation intervals with the same start time, without interrupting video playback. Note that these intervals are independant, unlike the intervals you create using the Alt key; you can end them at different times. This function is also useful if you erroneously start logging the wrong concept: instead of cancelling it, skipping back to the starting point, and starting logging of the correct concept, you can Ctrl-click on the right one, and then cancel the logging of the wrong one, thus correcting your mistake without interrupting video playback.
You can select an interval by clicking on it with the mouse. Its endpoints will be marked by vertical line segments, to indicate that it is selected. Note that you can only select "un-aggregated" intervals you may need to expand the tree list on the left to get to the interval you are interested in.
When you have selected an interval, you can change its endpoints by dragging them. You can also move the interval to a different concept by dragging it vertically. If you hold down the Alt key while dragging the interval, it will not be moved, but instead related to the concept you drop it on in addition to the original concept. If you hold down Ctrl, a copy of the interval will be created, related to the concept you drop it on.
Intervals can be split, using the Split button (Alt-s) in the toolbar. The selected interval is split at the current playback time, if this is between the interval's endpoints; otherwise, it is split in the middle. Intervals can be joined by dragging their endpoints so that they overlap. Note that this does not work for intervals related to more than one concept; this is intentional. Finally, you can delete intervals by selecting them and pressing the Delete key.
Intervals can also have properties, just like projects, media resources, ontologies and concepts. Interval properties are shown and changed in a separate window, which you can open by pressing the third button from the left on the toolbar. The intervals have a strange kind of generated URI, as it would be a horrible hassle for the user to have to come up with a unique URI for every interval. Intervals with properties are indicated in the Logging panel by a small icon by the left endpoint, but this is shown only when the interval is unaggregated you may have to expand the ontology tree on the left to see it.
You can zoom the interval display in and out using the two zoom buttons on the toolbar. The button to the right of them makes the display fit the width of the window, so you can see the whole thing without scrolling. You can also zoom by dragging the area you want to zoom into, on the time ruler.
If you have questions about OntoLog, do not hesitate to email me (address at the bottom of the page) or the OntoLog mailing list see http://groups.yahoo.com/group/ontolog.
1:
To be a bit more precise for the benefit of those who know RDFS: Classes are rdf:type rdfs:Class, and members are rdf:type their parent concept(s).
©2002-2005 Jon Heggland
jon.heggland + @ + idi.ntnu.no