[Blip.tv] [YouTube] [Vimeo]

I presented my final project for Intro to Computational Media today.  Building on the work I did with the Musical Typewriter, I ended up making some last minute tweaks to my “musical typeface,” which consists of audio samples corresponding to each letter of the alphabet.  Originally I had used all single-hit percussion sounds, but I reworked the alphabet to include short musical phrases or gestures, and brought in wind and string instrument samples for greater musicality and richness.  The final version of my Processing program reads a text file and “translates” the text into music by playing back the samples corresponding to the letters of the words as musical phrase “cluster.”

While playing back a text as a song, the Processing sketch also simultaneously displays the word corresponding to the musical sample cluster being played and visualizes the frequency waves of the music on the screen.  Refer to the screenshot above.

For my in-class performance, I used the lyrics to the HEPNOVA track Post-Modern Poster Child and reinterpreted words through the Musical Typewriter.  [MP3] [AIFF]

During my presentation, I also Rick Rolled the class with Rick Astley’s immortal lyrics played through my program.

Source code after the jump.

My source code for my ICM final project, a self-playing version of the Musical Typewriter [2] [3].

import ddf.minim.signals.*;
import ddf.minim.*;
import ddf.minim.analysis.*;
import ddf.minim.effects.*;
import processing.video.*;

// Declare MovieMaker object
//MovieMaker mm;
Minim minim;
WaveformRenderer waveform;
//AudioSample objects for each letter of the alphabet
AudioSample play_a;
AudioSample play_b;
AudioSample play_c;
AudioSample play_d;
AudioSample play_e;
AudioSample play_f;
AudioSample play_g;
AudioSample play_h;
AudioSample play_i;
AudioSample play_j;
AudioSample play_k;
AudioSample play_l;
AudioSample play_m;
AudioSample play_n;
AudioSample play_o;
AudioSample play_p;
AudioSample play_q;
AudioSample play_r;
AudioSample play_s;
AudioSample play_t;
AudioSample play_u;
AudioSample play_v;
AudioSample play_w;
AudioSample play_x;
AudioSample play_y;
AudioSample play_z;

// Variable to store text currently being typed
//String typing = “”;
String[] lyrics;
// Variable to store saved text when return is hit
String saved = “”;
int i;
int counter = 0;  //frame counter for vibration
int play = 0;

void setup()
{
minim = new Minim(this);
size(1440,800);
//mm = new MovieMaker(this, width, height, “pomo.mov” , 30, MovieMaker.H263,
//MovieMaker.HIGH);
background(0);
textFont(createFont(“Courier”, 60));
textAlign(CENTER);

play_a = minim.loadSample(“A.mp3″, 1440);
play_b = minim.loadSample(“B.mp3″, 1440);
play_c = minim.loadSample(“C.mp3″, 1440);
play_d = minim.loadSample(“D.mp3″, 1440);
play_e = minim.loadSample(“E.mp3″, 1440);
play_f = minim.loadSample(“F.mp3″, 1440);
play_g = minim.loadSample(“G.mp3″, 1440);
play_h = minim.loadSample(“H.mp3″, 1440);
play_i = minim.loadSample(“I.mp3″, 1440);

play_j = minim.loadSample(“J.mp3″, 1440);
play_k = minim.loadSample(“K.mp3″, 1440);
play_l = minim.loadSample(“L.mp3″, 1440);
play_m = minim.loadSample(“M.mp3″, 1440);
play_n = minim.loadSample(“N.mp3″, 1440);
play_o = minim.loadSample(“O.mp3″, 1440);
play_p = minim.loadSample(“P.mp3″, 1440);
play_q = minim.loadSample(“Q.mp3″, 1440);
play_r = minim.loadSample(“R.mp3″, 1440);
play_s = minim.loadSample(“S.mp3″, 1440);
play_t = minim.loadSample(“T.mp3″, 1440);
play_u = minim.loadSample(“U.mp3″, 1440);
play_v = minim.loadSample(“V.mp3″, 1440);
play_w = minim.loadSample(“W.mp3″, 1440);
play_x = minim.loadSample(“X.mp3″, 1440);
play_y = minim.loadSample(“Y.mp3″, 1440);
play_z = minim.loadSample(“Z.mp3″, 1440);

waveform = new WaveformRenderer();
play_a.addListener(waveform);
play_b.addListener(waveform);
play_c.addListener(waveform);
play_d.addListener(waveform);
play_e.addListener(waveform);
play_f.addListener(waveform);
play_g.addListener(waveform);
play_h.addListener(waveform);
play_i.addListener(waveform);
play_j.addListener(waveform);
play_k.addListener(waveform);
play_l.addListener(waveform);
play_m.addListener(waveform);
play_n.addListener(waveform);
play_o.addListener(waveform);
play_p.addListener(waveform);
play_q.addListener(waveform);
play_r.addListener(waveform);
play_s.addListener(waveform);
play_t.addListener(waveform);
play_u.addListener(waveform);
play_v.addListener(waveform);
play_w.addListener(waveform);
play_x.addListener(waveform);
play_y.addListener(waveform);
play_z.addListener(waveform);

noStroke();
lyrics = loadStrings(“lyrics.txt”);
i = 0;
background(0);
}

void draw()
{
if (play == 1){
counter = (counter+1)%25;
noStroke();
fill(255);
//text (lyrics[i],width/2, height/2);
//mm.addFrame();
if (counter == 0) {
if (i < lyrics.length)
{
background(0);
text (lyrics[i],width/2, height/2);
int wordLength = lyrics[i].length();
for (int j=0; j < wordLength; j++)
{
for(int k=0; k < wordLength; k++)
{
char letter = lyrics[i].charAt(k);
playLetter(letter);
waveform.draw();
}
}
waveform.draw();
i++;
}
else {
background(0);
//mm.finish();
//noLoop();
}
}
}
}

void playLetter (char letter_)
{
if ((letter_ == ‘a’) || (letter_ == ‘A’))
{
play_a.trigger();
}

if ((letter_ == ‘b’) || (letter_ == ‘B’))
{
play_b.trigger();
}

if ((letter_ == ‘c’) || (letter_ == ‘C’))
{
play_c.trigger();
}

if ((letter_ == ‘d’) || (letter_ == ‘D’))
{
play_d.trigger();
}

if ((letter_ == ‘e’) || (letter_ == ‘E’))
{
play_e.trigger();
}

if ((letter_ == ‘f’) || (letter_ == ‘F’))
{
play_f.trigger();
}

if ((letter_ == ‘g’) || (letter_ == ‘G’))
{
play_g.trigger();
}

if ((letter_ == ‘h’) || (letter_ == ‘H’))
{
play_h.trigger();
}

if ((letter_ == ‘i’) || (letter_ == ‘I’))
{
play_i.trigger();
}

if ((letter_ == ‘j’) || (letter_ == ‘J’))
{
play_j.trigger();
}

if ((letter_ == ‘k’) || (letter_ == ‘K’))
{
play_k.trigger();
}

if ((letter_ == ‘l’) || (letter_ == ‘L’))
{
play_l.trigger();
}

if ((letter_ == ‘m’) || (letter_ == ‘M’))
{
play_m.trigger();
}

if ((letter_ == ‘n’) || (letter_ == ‘N’))
{
play_n.trigger();
}

if ((letter_ == ‘o’) || (letter_ == ‘O’))
{
play_o.trigger();
}

if ((letter_ == ‘p’) || (letter_ == ‘P’))
{
play_p.trigger();
}

if ((letter_ == ‘q’) || (letter_ == ‘Q’))
{
play_q.trigger();
}

if ((letter_ == ‘r’) || (letter_ == ‘R’))
{
play_r.trigger();
}

if ((letter_ == ‘s’) || (letter_ == ‘S’))
{
play_s.trigger();
}

if ((letter_ == ‘t’) || (letter_ == ‘T’))
{
play_t.trigger();
}

if ((letter_ == ‘u’) || (letter_ == ‘U’))
{
play_u.trigger();
}

if ((letter_ == ‘v’) || (letter_ == ‘V’))
{
play_v.trigger();
}

if ((letter_ == ‘w’) || (letter_ == ‘W’))
{
play_w.trigger();
}

if ((letter_ == ‘x’) || (letter_ == ‘X’))
{
play_x.trigger();
}

if ((letter_ == ‘y’) || (letter_ == ‘Y’))
{
play_y.trigger();
}

if ((letter_ == ‘z’) || (letter_ == ‘Z’))
{
play_z.trigger();
}
}

void stop()
{
play_a.close();
minim.stop();
super.stop();
}

class WaveformRenderer implements AudioListener
{
private float[] left;
private float[] right;

WaveformRenderer()
{
left = null;
right = null;
}

synchronized void samples(float[] samp)
{
left = samp;
}

synchronized void samples(float[] sampL, float[] sampR)
{
left = sampL;
right = sampR;
}

synchronized void draw()
{
// we’ve got a stereo signal if right or left are not null
if ( left != null && right != null )
{
noFill();
stroke(255);
beginShape();
for ( int i = 0; i < left.length; i++ )
{
vertex(i, height/4 + left[i]*120);
}
endShape();
beginShape();
for ( int i = 0; i < right.length; i++ )
{
vertex(i, 3*(height/4) + right[i]*120);
}
endShape();
}
else if ( left != null )
{
noFill();
stroke(255);
beginShape();
for ( int i = 0; i < left.length; i++ )
{
vertex(i, height/2 + left[i]*120);
}
endShape();
}
}
}

void keyPressed(){
play = 1;
}

Source code after the jump:

import ddf.minim.signals.*;
import ddf.minim.*;
import ddf.minim.analysis.*;
import ddf.minim.effects.*;

Minim minim;
WaveformRenderer waveform;
//AudioSample objects for each letter of the alphabet
AudioSample play_a;
AudioSample play_b;
AudioSample play_c;
AudioSample play_d;
AudioSample play_e;
AudioSample play_f;
AudioSample play_g;
AudioSample play_h;
AudioSample play_i;
AudioSample play_j;
AudioSample play_k;
AudioSample play_l;
AudioSample play_m;
AudioSample play_n;
AudioSample play_o;
AudioSample play_p;
AudioSample play_q;
AudioSample play_r;
AudioSample play_s;
AudioSample play_t;
AudioSample play_u;
AudioSample play_v;
AudioSample play_w;
AudioSample play_x;
AudioSample play_y;
AudioSample play_z;

void setup()
{
minim = new Minim(this);
size(500,500);

background(0);
textFont(createFont(“Courier”, 50));
text(“Type letters”, 50, height/2-30);
text(“to play”, 50, height/2+45);

play_a = minim.loadSample(“A.mp3″, 512);
play_b = minim.loadSample(“B.mp3″, 512);
play_c = minim.loadSample(“C.mp3″, 512);
play_d = minim.loadSample(“D.mp3″, 512);
play_e = minim.loadSample(“E.mp3″, 512);
play_f = minim.loadSample(“F.mp3″, 512);
play_g = minim.loadSample(“G.mp3″, 512);
play_h = minim.loadSample(“H.mp3″, 512);
play_i = minim.loadSample(“I.mp3″, 512);
play_j = minim.loadSample(“J.mp3″, 512);
play_k = minim.loadSample(“K.mp3″, 512);
play_l = minim.loadSample(“L.mp3″, 512);
play_m = minim.loadSample(“M.mp3″, 512);
play_n = minim.loadSample(“N.mp3″, 512);
play_o = minim.loadSample(“O.mp3″, 512);
play_p = minim.loadSample(“P.mp3″, 512);
play_q = minim.loadSample(“Q.mp3″, 512);
play_r = minim.loadSample(“R.mp3″, 512);
play_s = minim.loadSample(“S.mp3″, 512);
play_t = minim.loadSample(“T.mp3″, 512);
play_u = minim.loadSample(“U.mp3″, 512);
play_v = minim.loadSample(“V.mp3″, 512);
play_w = minim.loadSample(“W.mp3″, 512);
play_x = minim.loadSample(“X.mp3″, 512);
play_y = minim.loadSample(“Y.mp3″, 512);
play_z = minim.loadSample(“Z.mp3″, 512);

waveform = new WaveformRenderer();
play_a.addListener(waveform);
play_b.addListener(waveform);
play_c.addListener(waveform);
play_d.addListener(waveform);
play_e.addListener(waveform);
play_f.addListener(waveform);
play_g.addListener(waveform);
play_h.addListener(waveform);
play_i.addListener(waveform);
play_j.addListener(waveform);
play_k.addListener(waveform);
play_l.addListener(waveform);
play_m.addListener(waveform);
play_n.addListener(waveform);
play_o.addListener(waveform);
play_p.addListener(waveform);
play_q.addListener(waveform);
play_r.addListener(waveform);
play_s.addListener(waveform);
play_t.addListener(waveform);
play_u.addListener(waveform);
play_v.addListener(waveform);
play_w.addListener(waveform);
play_x.addListener(waveform);
play_y.addListener(waveform);
play_z.addListener(waveform);

textFont(createFont(“Courier”, 200));
noStroke();
frameRate(30);
}

void draw()
{
noStroke();
fill(0,0,0,20);
rect(0,0,width,height);
waveform.draw();
}

void keyPressed()
{
fill(255);
if ( key == ‘a’)
{
play_a.trigger();
text(“A”, width/2-60, height/2+50);
}

if ( key == ‘b’)
{
play_b.trigger();
text(“B”, width/2-60, height/2+50);
}

if ( key == ‘c’)
{
play_c.trigger();
text(“C”, width/2-60, height/2+50);
}

if ( key == ‘d’)
{
play_d.trigger();
text(“D”, width/2-60, height/2+50);
}

if ( key == ‘e’)
{
play_e.trigger();
text(“E”, width/2-60, height/2+50);
}

if ( key == ‘f’)
{
play_f.trigger();
text(“F”, width/2-60, height/2+50);
}

if ( key == ‘g’)
{
play_g.trigger();
text(“G”, width/2-60, height/2+50);
}

if ( key == ‘h’)
{
play_h.trigger();
text(“H”, width/2-60, height/2+50);
}

if ( key == ‘i’)
{
play_i.trigger();
text(“I”, width/2-60, height/2+50);
}

if ( key == ‘j’)
{
play_j.trigger();
text(“J”, width/2-60, height/2+50);
}

if ( key == ‘k’)
{
text(“K”, width/2-60, height/2+50);
play_k.trigger();
}

if ( key == ‘l’)
{
play_l.trigger();
text(“L”, width/2-60, height/2+50);
}

if ( key == ‘m’)
{
text(“M”, width/2-60, height/2+50);
play_m.trigger();
}

if ( key == ‘n’)
{
play_n.trigger();
text(“N”, width/2-60, height/2+50);
}

if ( key == ‘o’)
{
play_o.trigger();
text(“O”, width/2-60, height/2+50);
}

if ( key == ‘p’)
{
play_p.trigger();
text(“P”, width/2-60, height/2+50);
}

if ( key == ‘q’)
{
play_q.trigger();
text(“Q”, width/2-60, height/2+50);
}

if ( key == ‘r’)
{
play_r.trigger();
text(“R”, width/2-60, height/2+50);
}

if ( key == ‘s’)
{
play_s.trigger();
text(“S”, width/2-60, height/2+50);
}

if ( key == ‘t’)
{
play_t.trigger();
text(“T”, width/2-60, height/2+50);
}

if ( key == ‘u’)
{
play_u.trigger();
text(“U”, width/2-60, height/2+50);
}

if ( key == ‘v’)
{
play_v.trigger();
text(“V”, width/2-60, height/2+50);
}

if ( key == ‘w’)
{
play_w.trigger();
text(“W”, width/2-60, height/2+50);
}

if ( key == ‘x’)
{
play_x.trigger();
text(“X”, width/2-60, height/2+50);
}

if ( key == ‘y’)
{
play_y.trigger();
text(“Y”, width/2-60, height/2+50);
}

if ( key == ‘z’)
{
play_z.trigger();
text(“Z”, width/2-60, height/2+50);
}

if (key == ‘ ‘)
{
fill(0,0,0,20);
noStroke();
rect(0,0,width,height);
}
}

void stop()
{
play_a.close();
minim.stop();
super.stop();
}

class WaveformRenderer implements AudioListener
{
private float[] left;
private float[] right;

WaveformRenderer()
{
left = null;
right = null;
}

synchronized void samples(float[] samp)
{
left = samp;
}

synchronized void samples(float[] sampL, float[] sampR)
{
left = sampL;
right = sampR;
}

synchronized void draw()
{
// we’ve got a stereo signal if right or left are not null
if ( left != null && right != null )
{
noFill();
stroke(255);
beginShape();
for ( int i = 0; i < left.length; i++ )
{
vertex(i, height/4 + left[i]*120);
}
endShape();
beginShape();
for ( int i = 0; i < right.length; i++ )
{
vertex(i, 3*(height/4) + right[i]*120);
}
endShape();
}
else if ( left != null )
{
noFill();
stroke(255);
beginShape();
for ( int i = 0; i < left.length; i++ )
{
vertex(i, height/2 + left[i]*120);
}
endShape();
}
}
}

ICM Final in Progress: Musical Typewriter

ICM Final Project Proposal: Text2Drum

After getting help from the ICM email list, I changed my AudioPlayer objects to AudioSamples, which only need to be triggered once, as opposed to AudioPlayer objects that require play and rewind functions.  This tweak solved my latency and crackle issues.

Testing out the new version has reminded me of the aural pleasure of typing on analog typewriters.  The rhythmic, percussive quality of clanking keys is now lost on contemporary computer keyboards that make more muted sounds when used.

I also thought it would be cool to visualize the waveform of the audio output and display that along with the letter on the screen.  Well, back to work…

New code after the jump:

import ddf.minim.signals.*;
import ddf.minim.*;
import ddf.minim.analysis.*;
import ddf.minim.effects.*;

Minim minim;
//AudioSample objects for each letter of the alphabet
AudioSample play_a;
AudioSample play_b;
AudioSample play_c;
AudioSample play_d;
AudioSample play_e;
AudioSample play_f;
AudioSample play_g;
AudioSample play_h;
AudioSample play_i;
AudioSample play_j;
AudioSample play_k;
AudioSample play_l;
AudioSample play_m;
AudioSample play_n;
AudioSample play_o;
AudioSample play_p;
AudioSample play_q;
AudioSample play_r;
AudioSample play_s;
AudioSample play_t;
AudioSample play_u;
AudioSample play_v;
AudioSample play_w;
AudioSample play_x;
AudioSample play_y;
AudioSample play_z;

void setup()
{
minim = new Minim(this);
size(500,500);

background(0);
textFont(createFont(“Courier”, 50));
text(“Type letters”, 30, height/2-50);
text(“to play”, 30, height/2+25);

play_a = minim.loadSample(“A.mp3″, 512);
play_b = minim.loadSample(“B.mp3″, 512);
play_c = minim.loadSample(“C.mp3″, 512);
play_d = minim.loadSample(“D.mp3″, 512);
play_e = minim.loadSample(“E.mp3″, 512);
play_f = minim.loadSample(“F.mp3″, 512);
play_g = minim.loadSample(“G.mp3″, 512);
play_h = minim.loadSample(“H.mp3″, 512);
play_i = minim.loadSample(“I.mp3″, 512);
play_j = minim.loadSample(“J.mp3″, 512);
play_k = minim.loadSample(“K.mp3″, 512);
play_l = minim.loadSample(“L.mp3″, 512);
play_m = minim.loadSample(“M.mp3″, 512);
play_n = minim.loadSample(“N.mp3″, 512);
play_o = minim.loadSample(“O.mp3″, 512);
play_p = minim.loadSample(“P.mp3″, 512);
play_q = minim.loadSample(“Q.mp3″, 512);
play_r = minim.loadSample(“R.mp3″, 512);
play_s = minim.loadSample(“S.mp3″, 512);
play_t = minim.loadSample(“T.mp3″, 512);
play_u = minim.loadSample(“U.mp3″, 512);
play_v = minim.loadSample(“V.mp3″, 512);
play_w = minim.loadSample(“W.mp3″, 512);
play_x = minim.loadSample(“X.mp3″, 512);
play_y = minim.loadSample(“Y.mp3″, 512);
play_z = minim.loadSample(“Z.mp3″, 512);

textFont(createFont(“Courier”, 200));
}

void draw()
{

}

void keyPressed()
{
if ( key == ‘a’)
{
background(0);
text(“A”, width/2-60, height/2+20);
play_a.trigger();
}

if ( key == ‘b’)
{
background(0);
text(“B”, width/2-60, height/2+20);
play_b.trigger();
}

if ( key == ‘c’)
{
background(0);
text(“C”, width/2-60, height/2+20);
play_c.trigger();
}

if ( key == ‘d’)
{
background(0);
text(“D”, width/2-60, height/2+20);
play_d.trigger();
}

if ( key == ‘e’)
{
background(0);
text(“E”, width/2-60, height/2+20);
play_e.trigger();
}

if ( key == ‘f’)
{
background(0);
text(“F”, width/2-60, height/2+20);
play_f.trigger();
}

if ( key == ‘g’)
{
background(0);
text(“G”, width/2-60, height/2+20);
play_g.trigger();
}

if ( key == ‘h’)
{
background(0);
text(“H”, width/2-60, height/2+20);
play_h.trigger();
}

if ( key == ‘i’)
{
background(0);
text(“I”, width/2-60, height/2+20);
play_i.trigger();
}

if ( key == ‘j’)
{
background(0);
text(“J”, width/2-60, height/2+20);
play_j.trigger();
}

if ( key == ‘k’)
{
background(0);
text(“K”, width/2-60, height/2+20);
play_k.trigger();
}

if ( key == ‘l’)
{
background(0);
text(“L”, width/2-60, height/2+20);
play_l.trigger();
}

if ( key == ‘m’)
{
background(0);
text(“M”, width/2-60, height/2+20);
play_m.trigger();
}

if ( key == ‘n’)
{
background(0);
text(“N”, width/2-60, height/2+20);
play_n.trigger();
}

if ( key == ‘o’)
{
background(0);
text(“O”, width/2-60, height/2+20);
play_o.trigger();
}

if ( key == ‘p’)
{
background(0);
text(“P”, width/2-60, height/2+20);
play_p.trigger();
}

if ( key == ‘q’)
{
background(0);
text(“Q”, width/2-60, height/2+20);
play_q.trigger();
}

if ( key == ‘r’)
{
background(0);
text(“R”, width/2-60, height/2+20);
play_r.trigger();
}

if ( key == ‘s’)
{
background(0);
text(“S”, width/2-60, height/2+20);
play_s.trigger();
}

if ( key == ‘t’)
{
background(0);
text(“T”, width/2-60, height/2+20);
play_t.trigger();
}

if ( key == ‘u’)
{
background(0);
text(“U”, width/2-60, height/2+20);
play_u.trigger();
}

if ( key == ‘v’)
{
background(0);
text(“V”, width/2-60, height/2+20);
play_v.trigger();
}

if ( key == ‘w’)
{
background(0);
text(“W”, width/2-60, height/2+20);
play_w.trigger();
}

if ( key == ‘x’)
{
background(0);
text(“X”, width/2-60, height/2+20);
play_x.trigger();
}

if ( key == ‘y’)
{
background(0);
text(“Y”, width/2-60, height/2+20);
play_y.trigger();
}

if ( key == ‘z’)
{
background(0);
text(“Z”, width/2-60, height/2+20);
play_z.trigger();
}

if (key == ‘ ‘)
{
background(0);
}
}

void stop()
{
minim.stop();
super.stop();
}

I have written a Processing sketch, with the help of the Minim library, that that plays back the “percussion letters” when the user types on the keyboard.  There is a bit of latency and audio “crackle” that still needs to be worked out, but for the most part, the musical typewriter works.  The next step is to build a related program that can read a text file as a musical score, translate the letters into percussion alphabet, play back the results, and save the audio playback as a file.

For more info, refer to my previous post about Text2Drum, or see my source code for the musical typewriter after the jump.

Musical Typewriter source code:

import ddf.minim.signals.*;
import ddf.minim.*;
import ddf.minim.analysis.*;
import ddf.minim.effects.*;

Minim minim;
//AudioPlayer objects for each letter of the alphabet
AudioPlayer play_a;
AudioPlayer play_b;
AudioPlayer play_c;
AudioPlayer play_d;
AudioPlayer play_e;
AudioPlayer play_f;
AudioPlayer play_g;
AudioPlayer play_h;
AudioPlayer play_i;
AudioPlayer play_j;
AudioPlayer play_k;
AudioPlayer play_l;
AudioPlayer play_m;
AudioPlayer play_n;
AudioPlayer play_o;
AudioPlayer play_p;
AudioPlayer play_q;
AudioPlayer play_r;
AudioPlayer play_s;
AudioPlayer play_t;
AudioPlayer play_u;
AudioPlayer play_v;
AudioPlayer play_w;
AudioPlayer play_x;
AudioPlayer play_y;
AudioPlayer play_z;

void setup()
{
minim = new Minim(this);
size(500,500);

background(0);
textFont(createFont(“Courier”, 50));
text(“Type letters”, 30, height/2-50);
text(“to play”, 30, height/2+25);

play_a = minim.loadFile(“A.mp3″, 512);
play_b = minim.loadFile(“B.mp3″, 512);
play_c = minim.loadFile(“C.mp3″, 512);
play_d = minim.loadFile(“D.mp3″, 512);
play_e = minim.loadFile(“E.mp3″, 512);
play_f = minim.loadFile(“F.mp3″, 512);
play_g = minim.loadFile(“G.mp3″, 512);
play_h = minim.loadFile(“H.mp3″, 512);
play_i = minim.loadFile(“I.mp3″, 512);
play_j = minim.loadFile(“J.mp3″, 512);
play_k = minim.loadFile(“K.mp3″, 512);
play_l = minim.loadFile(“L.mp3″, 512);
play_m = minim.loadFile(“M.mp3″, 512);
play_n = minim.loadFile(“N.mp3″, 512);
play_o = minim.loadFile(“O.mp3″, 512);
play_p = minim.loadFile(“P.mp3″, 512);
play_q = minim.loadFile(“Q.mp3″, 512);
play_r = minim.loadFile(“R.mp3″, 512);
play_s = minim.loadFile(“S.mp3″, 512);
play_t = minim.loadFile(“T.mp3″, 512);
play_u = minim.loadFile(“U.mp3″, 512);
play_v = minim.loadFile(“V.mp3″, 512);
play_w = minim.loadFile(“W.mp3″, 512);
play_x = minim.loadFile(“X.mp3″, 512);
play_y = minim.loadFile(“Y.mp3″, 512);
play_z = minim.loadFile(“Z.mp3″, 512);

textFont(createFont(“Courier”, 200));
}

void draw()
{

}

void keyPressed()
{
if ( key == ‘a’)
{
background(0);
text(“A”, width/2-60, height/2+20);
play_a.play();
play_a.rewind();
}

if ( key == ‘b’)
{
background(0);
text(“B”, width/2-60, height/2+20);
play_b.play();
play_b.rewind();
}

if ( key == ‘c’)
{
background(0);
text(“C”, width/2-60, height/2+20);
play_c.play();
play_c.rewind();
}

if ( key == ‘d’)
{
background(0);
text(“D”, width/2-60, height/2+20);
play_d.play();
play_d.rewind();
}

if ( key == ‘e’)
{
background(0);
text(“E”, width/2-60, height/2+20);
play_e.play();
play_e.rewind();
}

if ( key == ‘f’)
{
background(0);
text(“F”, width/2-60, height/2+20);
play_f.play();
play_f.rewind();
}

if ( key == ‘g’)
{
background(0);
text(“G”, width/2-60, height/2+20);
play_g.play();
play_g.rewind();
}

if ( key == ‘h’)
{
background(0);
text(“H”, width/2-60, height/2+20);
play_h.play();
play_h.rewind();
}

if ( key == ‘i’)
{
background(0);
text(“I”, width/2-60, height/2+20);
play_i.play();
play_i.rewind();
}

if ( key == ‘j’)
{
background(0);
text(“J”, width/2-60, height/2+20);
play_j.play();
play_j.rewind();
}

if ( key == ‘k’)
{
background(0);
text(“K”, width/2-60, height/2+20);
play_k.play();
play_k.rewind();
}

if ( key == ‘l’)
{
background(0);
text(“L”, width/2-60, height/2+20);
play_l.play();
play_l.rewind();
}

if ( key == ‘m’)
{
background(0);
text(“M”, width/2-60, height/2+20);
play_m.play();
play_m.rewind();
}

if ( key == ‘n’)
{
background(0);
text(“N”, width/2-60, height/2+20);
play_n.play();
play_n.rewind();
}

if ( key == ‘o’)
{
background(0);
text(“O”, width/2-60, height/2+20);
play_o.play();
play_o.rewind();
}

if ( key == ‘p’)
{
background(0);
text(“P”, width/2-60, height/2+20);
play_p.play();
play_p.rewind();
}

if ( key == ‘q’)
{
background(0);
text(“Q”, width/2-60, height/2+20);
play_q.play();
play_q.rewind();
}

if ( key == ‘r’)
{
background(0);
text(“R”, width/2-60, height/2+20);
play_r.play();
play_r.rewind();
}

if ( key == ‘s’)
{
background(0);
text(“S”, width/2-60, height/2+20);
play_s.play();
play_s.rewind();
}

if ( key == ‘t’)
{
background(0);
text(“T”, width/2-60, height/2+20);
play_t.play();
play_t.rewind();
}

if ( key == ‘u’)
{
background(0);
text(“U”, width/2-60, height/2+20);
play_u.play();
play_u.rewind();
}

if ( key == ‘v’)
{
background(0);
text(“V”, width/2-60, height/2+20);
play_v.play();
play_v.rewind();
}

if ( key == ‘w’)
{
background(0);
text(“W”, width/2-60, height/2+20);
play_w.play();
play_w.rewind();
}

if ( key == ‘x’)
{
background(0);
text(“X”, width/2-60, height/2+20);
play_x.play();
play_x.rewind();
}

if ( key == ‘y’)
{
background(0);
text(“Y”, width/2-60, height/2+20);
play_y.play();
play_y.rewind();
}

if ( key == ‘z’)
{
background(0);
text(“Z”, width/2-60, height/2+20);
play_z.play();
play_z.rewind();
}

if (key == ‘ ‘)
{
background(0);
}
}

I also hope to make a second version of Text2Drum which will be interactive, which will have an interface that will allow a user to type in text that Text2Drum will then convert into a rhythmic pattern.

The Portrait of Mao is a 42 inch by 42 inch color print on archival paper. The image was generated by a Processing sketch I wrote, which reads pixel color data from an image file and replaces each pixel with a text character with the same color as the original pixel. In the case of the Portrait of Mao, the source image was obtained from a Cultural Revolution-era LP cover of revolutionary songs featuring Chinese leader Mao Zedong and a bright, motley array of proletarian workers, representing different ethnic groups in China. My Processing sketch used this source image and replaced the pixels with the Chinese-language text of The Little Red Book AKA Quotations from Chairman Mao Zedong, a required text for every Chinese citizen during the Cultural Revolution.

In creating the Portrait, I wished to explore how text and images play a role in political propaganda and how they serve to construct a cult of personality around a paternal and pop-cultural icon. We know that words and images have persuasive and seductive powers, but how, and why? In reappropriating the propagandistic images and text and by depicting the controversial leader in an irreverently kitsch, Pop-Art way (Andy Warhol’s Mao paintings are an obvious point of reference), I sought to deconstruct the aura surrounding such a well-known figure who is still revered by millions in China.

The fusing of words and images also plays on the pictographic and ideographic nature of the graphemes used in the Chinese language. Chinese characters are simultaneously words, images, and symbols. In the Portrait of Mao they become both semantic and graphic building blocks of a text, an image, and an ideology.

Original blog post from 29 Oct 2008:

I presented my ICM midterm project yesterday.  I wanted to explore how text and images play a role in political propaganda and how they serve to construct a cult of personality.

The image above is just a scaled-down version, the real version is a 42 inch x 42 inch poster.  The poster consists of a text mosaic derived from a scan of a Cultural Revolution era LP cover of propaganda songs.  My Processing program reads the color information in the pixels of the source JPG and replaces it with characters from the Chinese version of The Little Red Book.

Here is a close up of the text mosaic, taken from the Technicolor Dream Coat guy on the bottom right.

I hope to one day present the poster in an over-the-top kitsch setting.  I want to frame it with red Christmas lights and build a “shrine” to the Chairman, with flowers and Tsing Dao beer bottles.  I also have another poster in the works featuring the Dear Leader composed out of the Korean-language text of his On the Juche Idea.  Ultimately, I want to create a triptych, but I haven’t decided on a third subject yet.  Uncle Ho?  Or maybe Grandpa Marx?

Here is a snapshot of me presenting the piece to my class.  Photo credit: Catherine White.

I started with the iconic image of Mao Zedong in Tiananmen Square.  Then I redrew the image using the text of the The Little Red Book.

Here is a close up:

I still have some work to do to address the spacing issues around the punctuation marks, which creates too much black space.  Ultimately, I intend to provoke a dialogue about the relationship between text and images, especially in the Chinese ideographic writing system, as well as explore the the roles that text and images play in political propaganda and the construction of ideology.

This week in Intro to Computational Media @ ITP, we learned how to add text into a Processing sketch.  I used my Sakura (cherry blossom) sketch from week 3 as a departure point and added the Japanese characters for sakura さくら into the new version of the sketch (above). I wanted to create a kind of “digital calligraphy” that explored the relationship between words and abstract shapes and set everything in motion. I also made an autumn leaves (koyo) sketch with the Japanese characters for koyo 紅葉 and an autumn color story.  To make things more visually appealing, I increased the size of the triangles and added some transparency to the background to create a more stylized sense of the passage of time in the animation and to create an illusion of three-dimensional depth.

There was a slight problem in getting the sketches to execute correctly though.  Although Processing is able to deal with Unicode-8 character sets, I was unable to get the Japanese characters to display correctly in the calligraphy font that I wanted.  The Japanese characters were showing, but in a default san-serif font and not the font that I created and specified in Processing.  I realized that I had to import the full character set and not just the default characters that Processing turns into bitmaps when you create a font.  However, when I checked “all characters” in the Create Font menu and clicked on “Create”, my computer froze up, probably because Japanese fonts have literally thousands of different characters, unlike the 26 letters and handful of punctuation marks we have in English.  Since I was only using 3 different characters in the Sakura sketch and 2 different characters in the Koyo sketch, I thought I that I might try creating SVG files in Adobe Illustrator for each character and then importing the Candy SVG library into my Processing sketches, which would then allow me to import the Japanese characters as vectors instead of bitmapped fonts.  Also, I only had to load the characters that I needed, and not the entire character set of the font.  As you can see from the screenshots, this approach worked!  I got the Japanese chracters to display in the calligraphy font instead of the default Processing font which didn’t work in the context of the sketches.

Click on the screenshots above and below to play with the sketches.  Drag the mouse around the frame and hold down any key on the keyboard to scatter cherry blossoms petals/autumn leaves.

Here is my abstract Processing version (or click on the screenshot below to view). Press any key or click the mouse in the square to scatter cherry blossom petals.

View source code

My drawing is a study of form, colors, and composition.  I used only circles, rectangles/squares and triangles.  I played with different degrees of color transparency (alpha) to layer shapes and colors.  As Scott McCloud pointed out in Understanding Comics (assigned reading in my Communications Lab class), circles/dots interact with the other random shapes to create eyes, faces, and breasts in the human mind, even in seemingly abstract drawings, which points to the anthro-centric hard-wiring of the human brain’s visual processing systems.

Click on the screenshot below to play with the interactive version.  Move your mouse around the drawing and see how it reacts.