Real-Instrument Synthesis

How do you synthesize the sound of real musical instruments? Synthesizers began as simple electronic oscillators, amplifiers and filters that sounded like radar equipment connected to a sound system. But there are plenty of composers who would like the real sounds of a symphony orchestra, or the real sounds of rock combo instruments. The technology of music synthesis has come a long way just since the Moog first appeared, and the keyboard synthesizers sold today have patches that sound incredibly real. Even though you can get the right sound with enough money, some of us would like to understand how to simulate real instruments.

The ideas here reflect my own point of view based on my own experimenting. I base my analysis on standard physics and signal processing theory, with some of my own interpretations. I assume the person reading this is a musician, not a mathematician. But in order to gain some insights into what I'm saying in the pages to follow you need to know or learn some math such as beginning calculus and physics. This subject is not easy, it takes a lot of hard work, and if you don't have time and patience then I suggest that you don't waste your time with this.

The short answer to the question of how do you synthesize real musical instruments is that you record them and play them back. I know, that's a rip-off answer! But that's how modern commercial synthesizers reproduce real-instrument sounds. The embarassment is assuaged by giving it the technical name, resampling. I shouldn't belittle this subject, because there is an art to resampling. Modern MIDI synthesizers break the sound into its beginning, middle and end so that you can hold a note as long as you want. This set of sound pieces taken together is commonly called a sound font. The Yamaha ES6 uses four overlapping sound pieces that Yamaha calls layers. Each layer is sent through its own effects channel before all four layers are mixed down to one output.

The first trick to resampling is to figure out how to loop the sound piece used for the middle part so that it sounds like one, long sound note. I've tried it. It takes hours of work to get one to sound close. There is a method for how you find the right starting point and ending point in the sound piece for your loop. You have to plot the waveform of the sound and stare at it with your eyeballs. You need to find a section that has as little amplitude (that is, envelope) variation as possible. You have to choose a section in which the waveform at the end looks very similar to the waveform at the beginning: you need to imagine that you're faking out the sampler into thinking that it is picking up where it left off, when in fact it is really starting over.

The second trick to resampling is putting some stereophonic effects into it. This makes a huge difference. A friendly hint: use Ecasound!

[ see Sound Resampling ]

The second most basic way to reproduce the real sound of a musical instrument is to use additive synthesis, that is the add together all its harmonics. If you knew what frequencies to use, you could use additive synthesis. But you need to know exactly what combination of harmonics to use. You can actually find that out by recording a sample and calculating its spectrum on your computer using an FFT program.

At this point, I know that I've left behind ninety percent of my readers. To do this you need to have some programming skill, some college-level mathematics knowledge, and be willing to put some time into it. But it's worse.

• You need to know how to calculate the frequency spectrum of the sound, that is, to use an FFT computer program
• You need to know how to analyse the frequency spectrum of the sound, that is, how to read the FFT.
• You need to use a computer program such as Scilab or Csound to create the sound using additive synthesis

Of all these hurtles, understanding the frequency spectrum is the most difficult. Even people who learned frequency analysis in college don't know how to do this. The question will arise what part of the spectrum is the beginning, middle and end of the sound? These questions are fundamental to certain sound engineering problems such as "stretching" a sound file. You'll be disappointed searching for this kind of information. At the time of this writing (May 2007) I have found absolutely no engineering information available, except for methods that don't work (linear predictive coding, granular synthesis), for sound stretching. I think I can help you with this, but you'll need to follow my discussions about frequency analysis.

Ultimately, using additive synthesis is not efficient. In practice, you have to add a great many frequencies to make the sound realistic. I have come close to reproducing the note of an acoustic piano after adding nearly a hundred frequency components. Instruments that have sharp attack envelopes, like the hammer impact on a piano, require many frequencies just to get the sharp beginning to sound right. Trying to compute a whole score by adding hundreds of frequencies for each note takes forever. The final, practical technique would be to simulate a few representitive notes, then use resampling to get the other notes in the scale.

At this point I will have lost all my readers. There are a handfull of people in the world who are interested in the actual physics of how musical instruments make there sounds.

Those of us who think this subject is important are perhaps disappointed that modern technology has made this subject irrevelant. You don't need physical modeling to acurately and beautifully reproduce the sound of a real musical instrument. Nothing sounds better than resampling, and if you don't know that yet then you haven't heard a modern high-priced keyboard synthesizer.

And yet a few of us think the knowledge is an end in itself, and it is likely that someday this knowledge will have a practical benefit.

Physical modeling is about using mathematics to predict how sound waves are produced, travel, reflect and resonate inside an object such as a musical instrument. There are no courses of study in college, that I know of, that are dedicated to this subject. This is mostly unexplored territory. Physics departments don't focus an acoustics because it is considered to be 19th century physics. There are a few electrical engineering departments that teach a brand of "acoustics" that is sorely inadequate, involving the discussion of helmholz resonators, compliances, inertances and waveguides. This kind of approach comes from the analogy of sound resonators to electrical resonators, and was a way to get some kind of answer in the days before computers (the helmholtz resonator model can approximate the lowest frequency of a resonator). Today, it's not worth the effort.

There is recently an interest in a particular kind of physical modeling, called Digital Waveguide Synthesis (DWS). Some writers just write "Physical Modeling," when they mean Digital Waveguide synthesis. If you read about some commercial synthesizers, say Yamaha, that use Physical Modeling, this will be Digital Waveguide synthesis.

The DWS technique is used to calculate the acoustics of a resonator by combining two computer-generated waves: a wave traveling forward and a wave traveling backward. I have heard a few nice-sounding instruments synthesized with this technique. The Csound music composition program has a violin instrument in its library that uses the DWS algorithm. I wrote a Csound piece using this instrument called sagittarius rex, if you want to hear what that sounds like.

I have read that there are numerous U.S. patents on DWS. I can't imagine what exactly is being patented. The superposition of a forward and backward traveling wave resulting in a standing wave is a physical fact, not an invention. There is, lately, an absurd idea that software algorithms qualify as inventions. Greed and ignorance. This is what happens when people are appointed to high offices (at the patent office) who lack the education needed to perform their duties.

Of all the physical modeling techniques, I am the least interested in the digital waveguide synthesis algorithm. It seems like a lot of work (and hype) over a result that can be created easier with a pencil and paper and a fourier series calculation. There is nothing easier to "model" than the sound of a plucked string. The mathematics for this has been known for at least a hundred years (see J.W.S. Rayleigh, The Theory of Sound, 1895). And in the end, the digital waveguide technique is severely limited. The phenomenon of two oppositely traveling waves combining into a standing wave is a special circumstance in one-dimension.

On the other hand, digital waveguide synthesis may prove to be useful as a nice trick for generating lots of harmonics with the least lines of code, in the same way that FM Synthesis has done. The concern in my mind is how much of this effort is really "physical" modeling? Aside from the basic premise of two reflected, traveling waves in a waveguide, most of this technique makes use of standard components from electrical engineering and signal processing such as filters, envelope generators, delay lines, etc. I presume, with a great deal of confidence, that the models that were created with this technique were had after much adjustment of parameters. At the end of the day, after spending many hours tweeking a sound to get it right, standing blurry-eyed at your office window, one wonders how much of the effort was "modeling" and how much of the effort was guessing. And perhaps this criticism may be applied to the entire subject of modeling.

• vibation modes (or FE)
• time-domain simulation (or ODE)

The most general way to model the vibrations of an object is by dividing the object into separate parts and calculating Newton's Second Law of motion on all the parts simultaneously. That method is called finite element modeling. It is very similar, and I would say equivalent, to numerically solving The Wave Equation.

vibration modes (or FE)

This has been done in aerospace engineering for years in designing aircraft. There is a classical technique, actually a step-by-step recipe, for creating a finite element model that is guaranteed to work if you do it right. The texts on this subject are hard to read, even for someone with a strong mathematics background. This is because the engineers who work in this area have created their own little kingdom with their own little nomenclature for their little recipe, not because there is anything difficult to understand (it's called job security). The essential idea is that the resonant frequencies of an object are calculated. For music synthesis, we would be calculating the resonant frequncies of, say, a guitar string or a drum head. The sound is then created using additive synthesis. The drawback of this technique is that in its most reliable form, the recipe mentioned above, it is a linear modeling technique. This will take some explaining when the time comes.

The aerospace companies spend thousands of dollars on software licenses for finite element software. But we can get there from here. I have a free finite element program, HMD for Computer Modeling that you can use. There are actually several free programs available, not just mine. I will use the program I wrote for my examples merely because I know the guy who wrote it.

time-domain simulation (or ODE)

The classical finite element technique does not directly calculate the time evolution of the vibrations, rather, the frequencies are found. But there is a way to directly simulate the time evolution. This technique doesn't really have a special name, except that control systems engineers have been using it for years to simulate feedback control behavior. It is referenced as numerical techniques for solving ordinary differential equations, or ODE. There are special algorithm names like, Euler, Runga-Kutta, Adams, Stiff Equations, etc. It will seem at first confusing to the trained mathematician that techniques for solving ordinary differential equations would be applied to spacial, boundary-value problems which have always been solved as partial differential equations. This works because the objects are subdivided into smaller finite elements that are each treated like oscillators coupled to their nearest neighbors.

The time domain method has the advantage that you can model non-linear problems. In the aerospace industry they spend thousands of dollars on software (i.e., Matlab) that can do ODE simulations. But we can get there from here. You can use the free software called Scilab, and for large, 3D models you'll write your own C code.

• The easiest way to synthesis real musical instruments is resampling, that is, recording the sound and then playing it back.
• Next, the most straightforward way is to add all the necessary harmonics. But we need to know exactlty which harmonics to use, and so we need to learn how to use an FFT program.
• Lastly, you can try physical modeling. Now you will be doing research, and you need to learn advanced mathematics, like how to solve differential equations.