Examples
From Synesketch Wiki
You can create wide range of possible applications and artworks with Synesketch. Basic use cases are presented bellow; yet, you can (and should) invent your own way of using it. To start, download Synesketch and install Java JRE 1.6 or higher. Also, we strongly suggest that you code in Eclipse, a superb programming environment.
Also, check out the Synesketch documentation. For more depth, read this paper (in Serbian).
Contents |
Importing Synesketch
To use and alter the Synesketch source code, open Eclipse and import Synesketch:
- File >> Import >> Existing Project into Workspace >> Browse >> find a Synesketch folder >> check project if not checked >> Finish
(If that doesn't work, try: File >> Import >> File System)
To use the Synesketch library, add synesketch.jar (library/synesketch.jar) into your project:
- Create a 'lib' folder in your workspace
- Paste synesketch.jar into that folder
- Right click to your project >> Build Path >> Configure Build Bath... >> Java Build Path >> Libraries >> Add Jars... >> find synesketch.jar in the lib folder of your project >> OK
Don't forget to do the same with the lib/processing/core.jar. And that's it.
Adding Synesketch to your app
This is the simplest way of using Synesketch. To add Synesketch visualizations to your app (e.g. a visual chat), you should look at the source code of the synesketch.app.Empathybox class. This is a simple class that provides a text field and a visual output. Adding Synesketch is basically two lines of code.
First, you should create an instance of the EmpathyPanel, a Swing component, subclass of a JPanel, which embeds the Processing applet (PApplet):
EmpathyPanel empathyPanel = new EmpathyPanel(400, "Synemania", "synesketch.emotion.SynesthetiatorEmotion");
Constructor parameters: dimension of the visualization; name of the visualization; full name of the synesthetiator type.
Second, you should inform a visualization (a Processing sketch) about the change in the text:
empathyPanel.fireSynesthesiator(text);
This should be a response to a certain event (e.g. "enter" is pressed during the chat conversation). For more insights, look at the source code of the synesketch.app.Empathybox class.
Creating Processing visualizations based on Synesketch
If you are a Processing artist/designer, we are really sorry to inform you that importing Synesketch into Processing won't work -- you'll have to use the Eclipse environment. Good news is that Eclipse is far more powerful and any serious development just isn't possible in the Processing environment. If you want to know more about the Processing-Eclipse relationship, read this and this. Changes you'll have to make to your Processing code are minor, but important ones.
Processing sketch in Java/Eclipse is simply a class which extends a PApplet class. When you create the class, you'll need to implement setup() and draw() methods. Running your visualization means running a plain old Java Applet:
- Right click on your class >> Run as >> Java Applet
Synesketch visualization is also a Processing sketch, with some small differences. First, you need to put your Synesketch sketch inside a synesketch.art.sketch package, in order to be available for applications which use that visualization!
Second. In order to use Synesketch for text visualizion, you need to define your response to a new text event in your Processing sketch (a class that extends PApplet). Practically, that means you have to implement only one extra method:
public void synesketchUpdate(SynesketchState state) {
//...
}
This method is being called when new text event occurs (e.g. new text entry within a chat conversation). The instance of the SynesketchState contains the important imformation about the text (e.g. emotional weights). It is up to you to decide in what way are you going to visualize this information.
We suggest that you take a look at the simple Hooloovoo visualization. It will give you a usefull insight about how the mechanism works. Here, we present the way this method was implemented in Hooloovoo:
public void synesketchUpdate(SynesketchState state) {
colorMode(HSB, 1.0f);
EmotionalState currentState = (EmotionalState) state;
Emotion emo = currentState.getStrongestEmotion();
int currentEmotion = emo.getType();
setSize(emo.getWeight());
saturation = (float) Math.sqrt(emo.getWeight());
setPallete(currentEmotion);
}
public void setPallete(int currentEmotion) {
if (currentEmotion == Emotion.NEUTRAL) {
currentPalette = bwPalette;
delay = 1500;
sat = 0.5f;
} else if (currentEmotion == Emotion.HAPPINESS) {
currentPalette = palette.getHappinessColors();
delay = 400;
} else if (currentEmotion == Emotion.SADNESS) {
currentPalette = palette.getSadnessColors();
delay = 1500;
} else if (currentEmotion == Emotion.ANGER) {
currentPalette = palette.getAngerColors();
delay = 300;
} else if (currentEmotion == Emotion.FEAR) {
currentPalette = palette.getFearColors();
delay = 800;
} else if (currentEmotion == Emotion.DISGUST) {
currentPalette = palette.getDisgustColors();
delay = 800;
} else if (currentEmotion == Emotion.SURPRISE) {
currentPalette = palette.getSurpriseColors();
delay = 200;
}
}
public void draw() {
for (int i = 0; i < dim/size + 1; i++) {
for (int j = 0; j < dim/size + 1; j++) {
int color = getRandomColor();
noStroke();
color = color(hue(color), saturation(color) * sat * 0.3f, brightness(color));
colorMode(RGB);
fill(red(color), green(color), blue(color), 1);
rect(i*size, j*size, size, size);
colorMode(HSB, 1.0f);
}
}
delay(delay);
}
As you can see, emotional types and weights are taken from the SynesketchState object casted into an EmotionalState object. This object contains emotinal parameters of a text: general emotional weight; general valence (emotion is positive or negative); six specific emotional weights; and a previous EmotionalState (so you can take into account the whole emotional history of one conversation). For more detailed explanation of a EmotionalState object, look at the "Using Synesketch for textual emotion sensing" section and a Synesektch documentation.
Emotional data is then being transferred into visualization parameters: color palette, saturation, and frame rate (delay). These parameters are being used by the draw() method to define a number of squares, their color pallete, size, and saturation.
Running a visualization
First of all, your visualization needs to be in a synesketch.art.sketch package.
To run and test your visualization you should use our demo app: Empathybox. Open the source code of the synesketch.app.Empathybox and change this line of code:
EmpathyPanel empathyPanel = new EmpathyPanel(dim, "Synemania", "synesketch.emotion.SynesthetiatorEmotion");
into this line:
EmpathyPanel empathyPanel = new EmpathyPanel(dim, "YourVisualizationName", "synesketch.emotion.SynesthetiatorEmotion");
YourVisualizationName is the name of your PApplet extending class -- of the Processing sketch of yours.
Now, run the application. That's it!
Using and creating color palettes
Synesketch has one standard six-part palette which defines palettes of colors for each of the six basic emotions: happiness, sadness, anger, fear, disgust, and surprise. This palette is based on the color psychology and design theories, yet this emotion-color mapping is highly subjective so we encourage you to define your own palettes.
Using the standard palette in your sketch means creating an instance of the SynesketchPalette class (see the documentation).
SynesketchPalette palette = new SynesketchPalette("standard");
Constructor parameter is the name of the palette. If you don't define your own, use "standard". Using colors is easy. For example, getting happiness-related colors looks like this:
int[] happinessColors = palette.getHappinessColors();
Now, if you want to create your own palette, take a look at the src/data/palette/standard.xml file. For each emotion, there is an array of colors defined by it's hex number. For example, this is the happiness-related color array:
<entry key="happiness.palette">e9d002, efd701, eccb00, eece25, f5ca01, f3c500, f9d646, efcb1f, f3cc29, fdbb03, f9ab01, f99f0b, fcae0e, f5a204, f07706, fb8933, fd812b, f94000, f85f26, f04c29, 701f1a, fa4c1a, f82301, cc0001, f31a11, ec0002, ed0101, eb0b15, df0303, c31455, c00000, 66a3e6, 5cbcd5, 54bdd3, 4cc949, 90dc00, d7dc00, cdbf00</entry>
To create your own palette, copy-paste this file, rename it, and alter colors/numbers.
Using Synesketch for textual emotion sensing
If not interested in visualizations, you can use Synesketch's textual emotion sensing engine aside. It uses a WordNet-based lexicon of words with emotional weights, and several NLP heuristic rules (e.g. emotional intensity of a sentence with emotional words is proportional to the number of exclamation marks at the end of the sentence). The algorithm transfers a (presumably small) chunk of text (e.g. basic unit of text in a chat conversation) into several emotional parameters. These parameters are encapsulated in a EmotionalState object (see the documentation), which defines emotional content of the text.
Parameters include:
- General emotional weight.
- General valence (emotion is positive or negative).
- Six specific emotional weights, defined by Ekman's categories: happiness weight, sadness weight, fear weight, anger weight, disgust weight, surprise weight. These specific emotions are defined by the class Emotion.
- Previous EmotionalState (so that whole emotional history of one conversation can be accessed from the Processing applet (PApplet)).
Weights have values between 0 and 1 (0 for no emotion, 1 for full emotion, 0.5 for the emotion of average intesity). Valence can be -1, 0, or 1 (negative, neutral, and positive emotion, respectively).
The following code presents how to acquire an instance of EmotionalState when having a chunk of text. It's exceptionally easy.
EmotionalState state = Empathyscope.getInstance().feel(text);
You can also alter a lexicon itself, which is in a form of a TXT file -- src/data/lex/synesketch_lexicon.txt -- or a XML one (soon to be added). Lexicon was formed automatically using WordNet, which is explained in detail in this paper (unfortunately still in Serbian) and will be explained in English in a paper that we are writing right now.
Creating synesthesia-like text-to-image tranformation apps
Textual emotion sensing is just one example of endless synesthesia-like transformations of texts to images. Using Synesketch, you can build your own text-to-image mapping algorithms. Design pattern-based Synesketch architecture, being very flexible and subsystems being completely mutually independent, makes that easy.
Particularly, Synesketch defines two abstract classes in a synesketch package:
- Synesthetiator, which defines common behavior for transferring textual information into visual output and notifying Processing applet (PApplet) about that new information. SynesthetiatorEmotion is a subclass of this class, where textual information refers to the new emotion recognized in text, which is to be transferred into visual patterns.
- SynesketchState, an abstract class which contains data textually interpreted by the Synesthetiator. It's subclass EmotionalState contains emotional information about the piece of text.
As you can see, textual emotion sensing is just one instance of this abstract textual interpretation and notification mechanism. Thus, you can create your own subclasses with different text interpretation algorithm. This algorithm should be implemented in a overridden synesthetise() method of a Synesthetiator abstract class. Data which is to be interpreted should be encapsulated in a subclass of a SynesketchState class. Look at the picture for the UML diagram representing this mechanism. Consult the documentation for details.
Following code illustrates this inheritance mechanism.
public class YourSynesthetiator extends Synesthetiator {
public SynesthetiatorEmotion(PApplet parent) throws Exception {
super(parent);
//...
}
@Override
public void synesthetise(String text) throws IOException {
//your text interpretation alghorithm in which
//you should create a SynesketchState out of the text...
YourSynesketchState currentState = yourInterpretationAlgorithm(text);
// now you should notify a Processing PApplet about that
notifyPApplet(currentState);
}
//...
}
public class YourSynesketchState extends SynesketchState {
public YourSynesketchState(String text) {
super(text);
//...
}
//...
}
When creating an EmpathyPanel in a GUI application (e.g. EmpathyBox), you should put your full Synesthetitor name (with packages!) as a third argument in an EmpathyPanel constructor:
EmpathyPanel empathyPanel = new EmpathyPanel(dim, "YourVisualizationName", "yourPackage.YourSynesthetiator");
If you need more information, feel free to contact us via mailing list.
